Network performance with VMXNET3 compared to E1000E and E1000. This article explains the difference between the virtual network adapters and part 2 will demonstrate how much network performance could be gained by selecting the paravirtualized adapter.
The VMware administrator has several different virtual network adapters available to attach to the virtual machines. The virtual adapters belong to either of two groups:
Emulated:
These are virtual hardware who emulates real existing physical network adapters. (Note that the physical network cards in the physical ESXi host is totally unrelated.) The VMkernel will present something that to the guest operating system will look exactly as some specific real world hardware and the guest could detect them through plug and play and use a native device driver.
Examples for the emulated devices are:
E1000 – which will emulate a 1 Gbit Intel 82545EM card, and is available for most operating systems since the generation of Windows Server 2003. This card is the default when creating almost all virtual machines and is by that widely used.
E1000E – emulates a newer real network adapter, the 1 Gbit Intel 82574, and is available for Windows 2012 and later. The E1000E needs VM hardware version 8 or later.
Above in Windows 2008 R2 with an emulated E1000 adapter the native guest operating system device driver is in use.
The positive side of the emulated network adapters are that they work “out of the box” and need no external code from VMware. It could be used to even (if needed) install the guest operating system by PXE since the E1000 device is available already from the BIOS start up.
The negative side is when using the default emulated adapters extra work is needed for every frame being sent or received from the guest operating system (which could be many thousands each second).
The VMkernel has to in “real time” emulate the exact behavior of the specific Intel 82545EM or 82574 cards, which will cost time and CPU cycles.
Paravirtualized:
The other type of virtual network adapters are the “paravirtualized”. The most recent one is called VMXNET3.
The paravirtualized network card does not exist as a physical NIC, but is a device “made up” entirely by VMware. For the guest operating system this will mean that it typically during the OS installation phase only senses that an unknown device is located in a PCI slot on the (virtual) motherboard, but it has no driver to actually use it.
(Note: some Linux distributions do even have the VMXNET3 driver pre-installed.)
For Windows Server, when a device driver is supplied, typically through the installation of VMware Tools, the guest operating system will perceive this as a real NIC from some network card manufacturer called “VMware” and use it as an ordinary network adapter. It has no reason to believe anything else than this is a NIC just as any other NIC around.
To the guest operating system the VMXNET3 card looks like a 10 Gbit physical device.
Note: there are also two obsolete paravirtualized adapters called VMXNET and VMXNET2 (sometimes the “Enhanced VMXNET”), however as long as the virtual machine has at least hardware version 7 only the VMXNET3 adapter should be used.
Since VMware with the VMXNET3 card owns much more of the network components even inside the VM there are many performance enhancements that could done. With the emulated E1000/E1000E the kernel has to mimic the exact behavior of existing adapters to the guest but with the VMXNET3 it could create a “perfect” virtual adapter optimized to be used in a virtual environment.
In part 2 of this article we will see how really large the performance difference actually is.
I simply dont understand. How paravirtualized network work when there is no Physical Adapter.
As Physical adapter responsibility to transmit/receive packets over Ethernet.
Hello Bilal,
and thank you for your question. The paravirtualized adapter could work totally without a physical adapter in the physical ESXi host, but you could in that case only transmit frames over the internal vSwitch.
No matter which virtual network card you attach to a virtual machine (E1000, E1000E, VMXNET3) it will a “fake” NIC that the VM belives is a real device, but is in reality a “soft” virtual adapter created by the VMkernel in CPU.
The physical adapter(s) is seen ONLY by ESXi Vmkernel and is only used when frames should leave the ESXi and travel to something on the outside.
Please return if you have further questions,
regards / Rickard
Thanks Rickard.
What I can understand that Physical adapter seen by VM Kernal and VM Kernal seen by VMNet Adapter.
For 10Gb/s I/O we should have Physical 10Gig Ethernet/ FCoE card otherwise VMNet3 dont work. Is that right?
Hello Bilal,
no it is not necessary to have physical 10 Gbit NIC in the ESXi host. The virtual machine will see a 10 Gbit (virtual) NIC attached to a virtual switch inside the host, but the VM will never see the physical interfaces which could be of any speed.
Regards, Rickard
Thanks Rickard,
So this means there is no interaction of Physical NIC(i.e 1Gbit) on the Esxi Host with the VMs having Virtual NIC VMNET3(10 Gbit).
Quite confused how its work as I/O via physical NIC(1 Gbit), How on Ethernet it works 10Gb.
Cool!
Bilal,
Think of those virtual Nic’s are on a virtual switch on the ESX host like Rickard said. If there are multiple VM’s on that ESX host, they would talk to each other at 10Gb within that virtual switch. The physical Nic on the host you would think of as WAN connection it’s bandwidth is limited based on it’s physical connection to the external switch. You would not throttle the bandwidth between the 2 VM’s because the physical NIC on the Host is 1Gb, you would want it at the maximum the OS can handle. If the Vm’s are on different hosts then it will transmit based on the physical bandwidth of the hosts.
Regards,
Mike
Thanks Mike. Thats what I am asking.
Hello Rickard,
I mentioned/linked your article in one of my articles today.
https://joeyverlinden.wordpress.com/2016/04/08/delayed-write-failed-windows-was-unable-to-save-all-the-data-uncnetwork-traffic/
Regards,
Joey
What about Bridge Network? Really need a good document to learn about it.
There are no “bridged” networks in the ESXi product, but it does exist in for example VMware Workstation.
Has anyone ever seen network congestion when VMXNET 3 is being used? We have a blade VM environment and use the e1000/e adaptor as its limited to 1Gb. Our concern is if all VM’s have 10Gb to the network that it could course congestion.
Thoughts?
No – never seen problems when using VMXNET3. You want to go as fast as you can. You are probably having latency issues (that you may not be aware of) if you are still using E1000. Thanks, -T
If you look at article 2 in this series you’ll notice that the virtual E1000e can go beyond 1Gb, so it won’t be providing the throttling that you expect. You would get better overall performance by using the VMXNET3 as the resource requirements for this NIC are lower – from this article “when using the default emulated adapters extra work is needed for every frame being sent or received from the guest operating system (which could be many thousands each second).”
Excellent thanks.
Hello Rickard,
good written article. I was just looking for improvments in network throughput for a W2K3R2 server on ESXi, which has to be in operation a little bit longer. Up to now it has the E1000, but i will try the VMXNet3. But first is to read part 2 of the article.
Karl
Hello Rickard,
On the moment I’am reinstalling my old vmware vSphere server 5.1 with some ‘old’ Novell Netware versions.
I run into the following problem, these are not existing with the installation of any nt4 to 2019 servers from our friend Ms.
If I install netware 4.11 I’dont see any network adaptor, everything is working fine except the connection to my clients.
In my server there is a Intel e1000 netwerkadaptor and that one cannot be found by any novell netware 3,4,5 and 6 versions.
I’ve tried the pcntnw.lan but that does not work to. Can you tell me what i should do ant install to get these server running as well ?
If you got a sollution for me that would be great, if not no harm done and i got to search on.
Thanks in advance,
Cees
Hello Cees,
when using really legacy operating systems, I would suggest testing the “VLANCE” virtual network adapter. This NIC is the highest chance for legacy system to detect.