Earlier this month I was experiencing problems with my servers. The symptom was all VoIP traffic was choppy. Turns out that the traffic was being sent just fine, it was delayed and when a phone receives delayed packets (or packets that are out of order) it drops that traffic so you don't hear garble.
This was a less than pleasant problem to figure out. However should you experience these problems, let this be one more place to look.
I decided it was time to take my VoIP servers and make them redundant. I only had two NICs on each machine, using one for private traffic and the other for public. Since Citrix XenServer supports the bonding of nics and 802.1q vlan tagging, I thought that this would make a good combination for HA, in the event of a NIC or switch failure.
So away I went. Bonding is fairly simple in Citrix XenCenter. You just tell it you want to create a bond, select the nics and in a few seconds you have a bond. Great, just create the networks and vlan then life is all peachy.
I thought I was all good until the next day I started to receive calls that voice was choppy. Let me preface this with, the bonding wasn't the only thing I changed, but were focusing on it so I'll only make reference to it. After many packet captures and testing I found that packets were not being lost, they were being delayed by about three seconds.
At this point I started to investigate how citrix xenserver bonds their networks. It turns out they use the standard linux kernel modules to do the bonding. This is very good. The bad part is, they choose to only support 'balance-slb' which isn't well documented at all. (If you find documentation on this type of load balancing, please comment below.) The main idea with this type of load balancing is to share the load over both links. They do this by changing what interface is used ever 10-30 seconds. (I can't get a strait answer from Citrix on the time. One engineer said 10 the other said 30 seconds) Well that sounds all fine and dandy until you checkout the default settings for the bond: