SR-IOV Network I/O Enhancement in a Virtualized Environment in Windows Server 2012

Betriebssysteme und Anwendungen

Betriebssysteme und Anwendungen
Betriebssysteme und Anwendungen

Betriebssysteme und Anwendungen - Wiki

SR-IOV Network I/O Enhancement in a Virtualized Environment in Windows Server 2012

Betriebssysteme und Anwendungen - Wiki

This post was originally written by Abhijit Khande & Vineeth V Acharya from DELL Windows Engineering Team.

Comments are welcome! To suggest a topic or make other comments, contact WinServerBlogs@dell.com.

With the Microsoft® Windows Server® 2012 Beta operating system, Microsoft has introduced support for a number of features in the networking space. One such significant and interesting feature is Single Root I/O Virtualization (SR-IOV), which allows virtual machines to share a single PCI-e device. This post provides a basic overview of SR-IOV in Windows Server 2012 Beta and how it drastically reduces the CPU utilization in a virtualized environment during network I/O.

The SR-IOV interface is an extension to the PCI Express (PCIe) specification. SR-IOV allows a single PCIe device, such as a network adapter, to provide multiple light weight hardware surfaces  on the PCI bus and segregate access to its resources among the various instances. It is achieved by the use of multiple virtual functions (VF) in addition to the (usual) physical function (PF). Since it is hardware dependent, the PCIe device and the platform must both support this feature.Here is the list of Dell PowerEdge™ platforms which support SR-IOV.

Traditionally, a packet destined for the virtual machine is received by the physical network adapter (using Physical Function) present in the host OS. This packet is handled by the NDIS driver module. The packet is then provided to the Hyper-V switch, which processes the packet (such as routing, VLAN filtering) and forwards the packet to the destined VM via VMBus, as shown in Figure-1. The reverse path will be followed when the VM has to send a packet.

 With SR-IOV, the VM can use a Virtual Function (VF) to send and receive packets directly to the physical network adapter bypassing the traditional path completely as shown in Figure-1. This not only boosts the network I/O but also reduces the overhead on the host machine’s CPU.

Live Migration & SR-IOV

In Windows Server 2012 Beta, Live Migration can be performed with SR-IOV being used by a VM. If the source and target systems support SR-IOV and the target has an available VF, the VM will use the virtual function. If not, the VM will revert to the traditional path (VM-Bus).

Each SR-IOV capable network adapter exposes a fixed number of Virtual Functions, which can be obtained by running the PowerShell command “Get-NetAdapterSriov”.


Performance Analysis

We performed a few tests in our lab to compare the performance with and without SR-IOV. The test environment consists of one test server (on which the tests are executed), and one file server. The test server is a Dell PowerEdge™ R710 II with an Intel® X520 10GB Ethernet adapter and Windows Server 2012 Beta OS. The file server hosts multiple SMB shares and is connected to the test server using a 10GB network via a Dell PowerConnect™ 8024 switch.

We captured the performance data in terms of the CPU used by the DPCs (Deferred Procedure Call) scheduled by the different driver modules taking part in the network data transfer. This data was captured both in the guest and host OS, as described in the following scenarios.The test server has 4 identical virtual machines with Windows Server 2012 Beta as the Guest OS. The 4 VMs are connected to the 10GB network via a virtual switch. This test configuration is shown in Figure 2.

 

Before we showcase the performance data, it is important to introduce a very critical parameter used for this testing. In the Microsoft Windows OS there exists a system mechanism called Deferred Procedure Call (DPC), which allows high-priority tasks (e.g. an interrupt handler) to defer required-but-lower-priority tasks for later execution. This permits device drivers and other low-level event consumers to perform the high-priority part of their processing quickly, and schedule non-critical additional processing for execution at a lower priority.

Scenario 1 – Performance Results in Virtual Machine (Guest OS):

We used four virtual machines (VM) for this scenario (VM 1 through VM 4). We enabled SR-IOV on VM 1 and VM 2 (by checking the “Enable SR-IOV” option in VM settings), and disabled SR-IOV on VM 3 and VM 4 (by unchecking the same option). Therefore, VM 1 and VM 2 will use the Virtual Function (exposed by the Intel adapter), while VM 3 and VM 4 will use the synthetic path (VMBus) for any network communication, as shown in Figure-2.

We started copying data (of size 20GB) from a single SMB share to the VMs, and captured the system CPU Usage logs in the Guest OS.

In Figure 3, SR-IOV refers to the average DPC CPU usage in VM-1 and VM-2 and Non-SR-IOV refers to the average DPC CPU usage in VM-3 and VM-4. If we look at the graph, we see that there is not much difference in the CPU usage between the two cases, but we haven’t accounted for the CPU Usage in the host machine yet.

Scenario 2 – Performance Result in Host Machine (Host OS):

We used one virtual machine for this scenario. We started copying data (of size 20GB) from the SMB share to the VM and captured the DPC CPU Usage of 2 modules, NDIS and VMBUS, which are used in network I/O.

 

The results are shown in Figure 4. As expected, there is a vast difference in the CPU usage between the two cases (SR-IOV and Non SR-IOV). The CPU usage is of the order of 102 in case of SR-IOV, and of the order 103 - 104 in case of Non SR-IOV. This is mainly because, in case of SR-IOV, the virtual machine directly communicates with the physical NIC via the virtual function. Hence, the CPU cycles of the host is not utilized for processing any network packets. In case of Non SR-IOV (as shown in Figure-1), the Guest OS communicates with the Host OS via the VM Bus, which in-turn processes the packets and sends it via the physical NIC. Hence, modules like VM Bus and NDIS are extensively used.

On calculating the total CPU Usage, we observe that the CPU utilization during network I/O is far less when using SR-IOV. Hence, Windows Server 2012 Beta with SR-IOV enabled will help customers reduce the CPU overhead during network I/O and thereby improve the overall system performance.

Configuring SR-IOV using PowerShell

Additionally, you could use the following PowerShell commands to create a new virtual switch with SR-IOV enabled and attach the virtual switch to the virtual network adapter of an existing virtual machine.

Note: Before running the following commands, the following options should be enabled in the BIOS

  • Virtualization Technology
  • SR-IOV Global Enable

 (Assuming a single network adapter (Intel X520) is connected)

$NetAdap = Get-NetAdapter | Where-Object { $_.Status -eq "Up"}

 

(The -EnableIov switch is used to enable SR-IOV in the virtual switch)

New-VMSwitch -Name "SRIOV Switch" -NetAdapterName $NetAdap.Name -AllowManagementOS $True -Notes "SRIOV Switch on X520" -EnableIov $True

 $VMSw = Get-VMSwitch

 

Adding a new network adapter to VM-1 and connecting it to the Virtual Switch.

Add-VMNetworkAdapter -SwitchName $VMSw.Name -VMName VM-1 -Name “SRIOV Adapter”

(Note: For this command to work, the VM must be turned off)

 

You can also use an already existing VM network adapter by using the following commands)

$VMNet = Get-VMNetworkAdapter -VMName VM-1

Connect-VMNetworkAdapter -VMName VM-1 -SwitchName $VMSw.Name -Name $VMNet.Name

 

Each VM network adapter has two properties, IovWeight & VmqWeight, which correspond to SR-IOV and VMQ respectively. Adjusting these weights enables or disables the features.

To enable SR-IOV, set the IovWeight to 100. To disable SR-IOV, set the IovWeight to 0 (default)

Set-VMNetworkAdapter -VMName VM-1 -VMNetworkAdapterName $VMNetName -IovWeight 100

where $VMNetName is the name of the VM Network Adapter connected to the SRIOV Switch.

 

Visit the following links for additional information:

0