On the vSphere virtual switch we have the possibility to enable or disable the option of “Notify Switches”. In this post we shall see what this setting actually does and how it works, as well as discussing if the RARP protocol is important.
To see the current setting or to change it, select Properties on a vSwitch, then Edit and then the NIC Teaming tab.
When having the value set to Yes it will basically give ESXi the permission to sometimes send faked frames on behalf of the Virtual Machines running on the host.
The goal for sending these frames is to make sure the physical switches in the network learns the location of the Virtual Machines. A physical switch does this learning by observing each incoming frame and make a note of the field called Source MAC Address. Based on that information the switches build tables with mappings between MAC addresses and the switch port where this address could be found.
There are at least three different occasions where these messages are sent by ESXi:
1. When a Virtual Machine is powered on the ESXi host will make sure the network are aware of this new VM and its MAC address as well as its physical location (switch port).
2. When a Virtual Machine is moved by vMotion to another host it will be very crucial to rapidly notify the physical network of the new placement of the virtual server.
3. If a physical NIC on the host loses the connection, but we had at least two vmnics as uplinks on the vSwitch all traffic from the VMs will be “moved” to the remaining interface. Also in this situation the ESXi host must very quickly inform the network of the new correct uplink to reach the VM.
The protocol used is RARP – Reverse Address Resolution Protocol. Sometimes in texts about vSphere it is implied that RARP itself is important and actually does something in this process. However, RARP is a very old and obsolete protocol which almost no operating systems or devices has support for today.
So why send frames that almost nobody could read? The reason is that it is only important that all physical switches sees the Source MAC address and learn the new location and in reality the payload (in this case RARP) could be almost anything.
Here we see a virtual machine with a certain MAC address. This VM is being migrated with vMotion and with Wireshark we shall see which RARP frames being sent.
In this packet trace we can notice that the ESXi host is sending RARP frames with the VM as source MAC and also that it wants to be very certain that every switch has been reached by this new information – and actually sends the frame ten times. First several times close together and then more spread in time, with the last frame almost one minute after the first information frame.
How could we be sure that all switches on the physical network does see these frames? This is because of the destination is the special broadcast address FF-FF-FF-FF-FF-FF. The broadcast address forces all switches to forward the frame on all ports and makes the whole broadcast domain aware of the new location.
If the Notify Switches settings for some reason is set to No then the behavior is changed and the host will be silent when actions like VM power-on, vMotion and vmnic link failure take place. It will likely still work, but a much higher risk of packet loss in the above situations.
In summary, the Notify Switches function is very useful and should be set to Yes to allow the ESXi host acting as the virtual machines and sends faked frames to make sure all physical switches MAC-to-port mapping tables are quickly updated.
Hi,
Really great information from you, now I got the very in depth theory of the RARP in ESXi. Thanks for your effort.
/ Gopi
We are likely facing the situation explined in – http://kb.vmware.com/kb/1556.
After seeing your excellent explanation of this functionality, I dont want to loose this, so is there a way that I can still leave this setting to Yes but some how instruct the host to use the bogus MAC address (in its RARP) for each NLB guest?
If I register static arp entries on the hosts for each NLB guest, will this force the hosts to use such bogus MAC in RARP? Any help would be much appreciated, thanks in advance.
Hello Mohan,
and thank you for your reply.
From my understanding of Microsoft NLB Clusters I think the key point is that the NLB machines (physical or virtual) depend on that switches never learn their true position and always floods every incoming frame to all destination ports, making sure this hits all the NLB machines.
Because of this it should be recommended to keep the NLB machines into their own VLAN to limit the frame flooding. If so you should be able to disable the “VMware Notify Switches” setting on these portgroups only, but keep the rest of your VM portgroups with the Notify setting enabled.
This is great information for me to understand why vMotion sends RARP packets after migration is over. But one question is that what if the server is connected with a Layer 3 switch/router directly ?
If RARP is not processed by switch/router (which is commonly seen now), this means that MAC:Port is updated by RARP packet but IP:MAC not. This could make influence on data traffic.
Thank you for your comment wildwill.
Does your question concerns having one ESXi host directly connected to a routing device with no layer two switching between and another ESXi on the remote side of the router?
For vMotion to actually work you would need to have a common layer two infrastructure, both for the vmkernel adapters with vMotion tag and for all Virtual Machine networks. That would mean that connecting two ESXi host to opposite sides of a router would not work with vMotion. (It could be done with different tunneling techniques, but not out of the box.)
As for the relation IP to MAC, typically done with classic ARP resolution, the Virtual Machine itself is not in any way changed in the vMotion process. This means the virtual network card will keep the same MAC address and the TCP/IP stack inside the guest operating system will have the same IP address.
The effect of that nothing changes inside the guest is that in a vMotion transfer there is also no need to change or update any existing ARP entries on any external devices.
Please return if something is unclear,
best regards, Rickard
I am having a problem where a vmotion completes successfully but the VM drop 4-6 pings instead of losing 1 or no pings. I have been troubleshooting this issue for several days and found your article helpful in explaining things, but I still have not solved the problem. I have the notify switches setting set on all hosts. I tried running wireshark on a vm that successfully vmotions with 1 or no ping drops (and vmotion that drops 4-6 pings) and I don’t see the rarp packets in either trace. All the ESXi logs seem to show the vMotion is successful, but for some reason the network switchover is slow. Any ideas?
Andrew, even I’m facing problem similar to what you’ve mentioned. Were you able to find the root cause?
Great explanation Rickard!
Thx!
I wonder why not to send a gratuitous ARP instead: much more supported, not obsolete, and makes the same effect.
Anyone knows?
Hello Pablo,
the main reason in my opinion is that it does not matter WHAT you send, the only thing important is that SOMETHING is sent with the hardware address of the VM as source MAC in the ethernet frame.
The gratuitous ARP is not really needed since there is no change in the relation between IP and MAC, and it might possible disturb more machines on the network since everyone will process ARP (ethertype 0806), but rarely RARP (8035).
Regards, Rickard
Great writing. I will read more. Subscribing.
Great explanation, thanks!
please any answer for this comment
I am having a problem where a vmotion completes successfully but the VM drop 4-6 pings instead of losing 1 or no pings. I have been troubleshooting this issue for several days and found your article helpful in explaining things, but I still have not solved the problem. I have the notify switches setting set on all hosts. I tried running wireshark on a vm that successfully vmotions with 1 or no ping drops (and vmotion that drops 4-6 pings) and I don’t see the rarp packets in either trace. All the ESXi logs seem to show the vMotion is successful, but for some reason the network switchover is slow. Any ideas?
Great article, though also experiencing this issue. Hope you can explain.
I am having a problem where a vmotion completes successfully but the VM drop 4-6 pings instead of losing 1 or no pings. I have been troubleshooting this issue for several days and found your article helpful in explaining things, but I still have not solved the problem. I have the notify switches setting set on all hosts. I tried running wireshark on a vm that successfully vmotions with 1 or no ping drops (and vmotion that drops 4-6 pings) and I don’t see the rarp packets in either trace. All the ESXi logs seem to show the vMotion is successful, but for some reason the network switchover is slow. Any ideas?
Thanks for your comment!
The RARP packets themselves will actually not be seen inside the VM who “sends” them, as it is the VMkernel that does this on its own, pretending to be the VM. If you place the Wireshark on some other port on the same physical VLAN you should be able to see them.
As for the general questions, it is unfortunately very difficult to give any advice due to the many factors of the whole configuration unknown.
Best regards, Rickard
This is a best document i have found related to topic, Someone refer me this link and found my problem solved easily.
Good article, thank you.