XenServer SR-IOV Support for Provisioning Services Virtual Machines
Created On: 10-Sep-2010
Updated On: 22-Dec-2010Average Rating:
As companies virtualize more and more of their server infrastructure they are increasingly interested in virtualizing Provisioning Services hosts. However, the significant level of network IO that occurs on a Provisioning Services host can be challenging for the virtualization platform. This article describes how to virtualize Provisioning Services hosts with maximum performance and scalability by leveraging Single Route I/O Virtualization (SR-IOV) capabilities of Intel Niantic NICs with XenServer (versions 5.6 and later).
SR-IOV is a PCI device virtualization technology that allows a single PCI device to appear as multiple PCI devices on the physical PCI bus: the real physical device is called physical function (PF) while the others are called virtual functions (VF).
The purpose of this is for the hypervisor to directly assign one or more of these VFs to a virtual machine using Intel VT-D technology: the guest can use the VF as any other directly assigned PCI device.
The common use case is an SR-IOV NIC. Assigning one or more VFs to a virtual machine allows the virtual machine to directly exploit the hardware without any mediation by the hypervisor. This means better performances and scalability because it has very little or no impact on dom0.
On the downside the VFs have no relations with VIFs and bridges in dom0 so they have to be configured separately and independently by the user. Also, a virtual machine loses all the assigned VFs after being migrated to a different host. Thus, virtual machines that are migrated between hosts using XenMotion or High Availability (HA) failover require manual reinstatement of VFs on the new host.
Because Provisioning Services hosts tend to be network I/O bound rather than memory or CPU bound, they are ideal candidates to take advantage of this capability. The limitations on XenServer virtual machine failover and XenMotion are not significant in a Provisioning Services deployment as Provisioning Services implements its own HA and load balancing mechanisms.
To make use of this capability, you must have a host server in which a SR-IOV capable network device is installed. The device tested for this article is the Intel® 82599 10 Gigabit Ethernet Controller.
Note: The setup procedure below requires that the 10 GigE NIC not be used as the management interface for the host. A second physical NIC must be installed on the system for that purpose.
The following procedure is a detailed step-by-step guide for configuring an SR-IOV-enabled Provisioning Services virtual machine on XenServer 5.6, provided that your system meets the hardware and firmware requirements described above.
Enable iommu on the host by running the following command within the domain 0 console:
/opt/xensource/libexec/xen-cmdline –set-xen iommu=1
Reboot the host. Assign one or more VFs to a virtual machine. Obtain a list of VFs in the system by executing the lspci command in the XenServer console. You should see many (such as 120) devices like this:
07:10.0 Ethernet controller: Intel Corporation 82559 Ethernet Controller Virtual Function (rev 01)
Assign one of these VFs to the target virtual machine executing the following command:
xe vm-param-set other-config:pci=0/0000:07:10.0 uuid=uuid_of_the_VM
Substitute 07:10.0 in the example above with the pci bus address of the VF you want to assign.
Boot the virtual machine and install the correct VF driver in it.
Once the driver is installed, Provisioning Services should be installed and configured as normal.
This should resolve the performance issues with 10GB ethernet & XenServer VM’s