On the vSphere virtual switch we have the possibility to enable or disable the option of “Notify Switches”. In this post we shall see what this setting actually does and how it works, as well as discussing if the RARP protocol is important.
To see the current setting or to change it, select Properties on a vSwitch, then Edit and then the NIC Teaming tab.
When having the value set to Yes it will basically give ESXi the permission to sometimes send faked frames on behalf of the Virtual Machines running on the host.
The goal for sending these frames is to make sure the physical switches in the network learns the location of the Virtual Machines. A physical switch does this learning by observing each incoming frame and make a note of the field called Source MAC Address. Based on that information the switches build tables with mappings between MAC addresses and the switch port where this address could be found.
There are at least three different occasions where these messages are sent by ESXi:
1. When a Virtual Machine is powered on the ESXi host will make sure the network are aware of this new VM and its MAC address as well as its physical location (switch port).
2. When a Virtual Machine is moved by vMotion to another host it will be very crucial to rapidly notify the physical network of the new placement of the virtual server.
3. If a physical NIC on the host loses the connection, but we had at least two vmnics as uplinks on the vSwitch all traffic from the VMs will be “moved” to the remaining interface. Also in this situation the ESXi host must very quickly inform the network of the new correct uplink to reach the VM.
The protocol used is RARP – Reverse Address Resolution Protocol. Sometimes in texts about vSphere it is implied that RARP itself is important and actually does something in this process. However, RARP is a very old and obsolete protocol which almost no operating systems or devices has support for today.
So why send frames that almost nobody could read? The reason is that it is only important that all physical switches sees the Source MAC address and learn the new location and in reality the payload (in this case RARP) could be almost anything.
Here we see a virtual machine with a certain MAC address. This VM is being migrated with vMotion and with Wireshark we shall see which RARP frames being sent.
In this packet trace we can notice that the ESXi host is sending RARP frames with the VM as source MAC and also that it wants to be very certain that every switch has been reached by this new information – and actually sends the frame ten times. First several times close together and then more spread in time, with the last frame almost one minute after the first information frame.
How could we be sure that all switches on the physical network does see these frames? This is because of the destination is the special broadcast address FF-FF-FF-FF-FF-FF. The broadcast address forces all switches to forward the frame on all ports and makes the whole broadcast domain aware of the new location.
If the Notify Switches settings for some reason is set to No then the behavior is changed and the host will be silent when actions like VM power-on, vMotion and vmnic link failure take place. It will likely still work, but a much higher risk of packet loss in the above situations.
In summary, the Notify Switches function is very useful and should be set to Yes to allow the ESXi host acting as the virtual machines and sends faked frames to make sure all physical switches MAC-to-port mapping tables are quickly updated.