RICKARD NOBEL AB

RICKARD NOBEL AB

Specialists in IT infrastructure services

Menu
  • About
  • Windows
  • Networking
  • VMware
  • Storage
Menu

The vSwitch “Notify Switches” setting

Posted on July 14, 2012September 29, 2014 by Rickard Nobel

On the vSphere virtual switch we have the possibility to enable or disable the option of “Notify Switches”. In this post we shall see what this setting actually does and how it works, as well as discussing if the RARP protocol is important.

notify-switches

To see the current setting or to change it, select Properties on a vSwitch, then Edit and then the NIC Teaming tab.

When having the value set to Yes it will basically give ESXi the permission to sometimes send faked frames on behalf of the Virtual Machines running on the host.

The goal for sending these frames is to make sure the physical switches in the network learns the location of the Virtual Machines. A physical switch does this learning by observing each incoming frame and make a note of the field called Source MAC Address. Based on that information the switches build tables with mappings between MAC addresses and the switch port where this address could be found.

There are at least three different occasions where these messages are sent by ESXi:

1. When a Virtual Machine is powered on the ESXi host will make sure the network are aware of this new VM and its MAC address as well as its physical location (switch port).

2. When a Virtual Machine is moved by vMotion to another host it will be very crucial to rapidly notify the physical network of the new placement of the virtual server.

notify-switches

3. If a physical NIC on the host loses the connection, but we had at least two vmnics as uplinks on the vSwitch all traffic from the VMs will be “moved” to the remaining interface. Also in this situation the ESXi host must very quickly inform the network of the new correct uplink to reach the VM.

The protocol used is RARP – Reverse Address Resolution Protocol. Sometimes in texts about vSphere it is implied that RARP itself is important and actually does something in this process. However, RARP is a very old and obsolete protocol which almost no operating systems or devices has support for today.

So why send frames that almost nobody could read? The reason is that it is only important that all physical switches sees the Source MAC address and learn the new location and in reality the payload (in this case RARP) could be almost anything.

notify-switches

Here we see a virtual machine with a certain MAC address. This VM is being migrated with vMotion and with Wireshark we shall see which RARP frames being sent.

notify-switches

In this packet trace we can notice that the ESXi host is sending RARP frames with the VM as source MAC and also that it wants to be very certain that every switch has been reached by this new information – and actually sends the frame ten times. First several times close together and then more spread in time, with the last frame almost one minute after the first information frame.

notify-switches

How could we be sure that all switches on the physical network does see these frames? This is because of the destination is the special broadcast address FF-FF-FF-FF-FF-FF. The broadcast address forces all switches to forward the frame on all ports and makes the whole broadcast domain aware of the new location.

If the Notify Switches settings for some reason is set to No then the behavior is changed and the host will be silent when actions like VM power-on, vMotion and vmnic link failure take place. It will likely still work, but a much higher risk of packet loss in the above situations.

In summary, the Notify Switches function is very useful and should be set to Yes to allow the ESXi host acting as the virtual machines and sends faked frames to make sure all physical switches MAC-to-port mapping tables are quickly updated.

25 thoughts on “The vSwitch “Notify Switches” setting”

  1. Gopikeerthy says:
    July 15, 2012 at 19:10

    Hi,

    Really great information from you, now I got the very in depth theory of the RARP in ESXi. Thanks for your effort.

    / Gopi

    Reply
  2. Mohan says:
    December 23, 2012 at 14:44

    We are likely facing the situation explined in – http://kb.vmware.com/kb/1556.

    After seeing your excellent explanation of this functionality, I dont want to loose this, so is there a way that I can still leave this setting to Yes but some how instruct the host to use the bogus MAC address (in its RARP) for each NLB guest?

    If I register static arp entries on the hosts for each NLB guest, will this force the hosts to use such bogus MAC in RARP? Any help would be much appreciated, thanks in advance.

    Reply
    1. Rickard Nobel says:
      December 23, 2012 at 15:28

      Hello Mohan,
      and thank you for your reply.

      From my understanding of Microsoft NLB Clusters I think the key point is that the NLB machines (physical or virtual) depend on that switches never learn their true position and always floods every incoming frame to all destination ports, making sure this hits all the NLB machines.

      Because of this it should be recommended to keep the NLB machines into their own VLAN to limit the frame flooding. If so you should be able to disable the “VMware Notify Switches” setting on these portgroups only, but keep the rest of your VM portgroups with the Notify setting enabled.

      Reply
  3. wildwill says:
    April 28, 2013 at 05:30

    This is great information for me to understand why vMotion sends RARP packets after migration is over. But one question is that what if the server is connected with a Layer 3 switch/router directly ?

    If RARP is not processed by switch/router (which is commonly seen now), this means that MAC:Port is updated by RARP packet but IP:MAC not. This could make influence on data traffic.

    Reply
    1. Rickard Nobel says:
      April 28, 2013 at 08:23

      Thank you for your comment wildwill.

      Does your question concerns having one ESXi host directly connected to a routing device with no layer two switching between and another ESXi on the remote side of the router?

      For vMotion to actually work you would need to have a common layer two infrastructure, both for the vmkernel adapters with vMotion tag and for all Virtual Machine networks. That would mean that connecting two ESXi host to opposite sides of a router would not work with vMotion. (It could be done with different tunneling techniques, but not out of the box.)

      As for the relation IP to MAC, typically done with classic ARP resolution, the Virtual Machine itself is not in any way changed in the vMotion process. This means the virtual network card will keep the same MAC address and the TCP/IP stack inside the guest operating system will have the same IP address.

      The effect of that nothing changes inside the guest is that in a vMotion transfer there is also no need to change or update any existing ARP entries on any external devices.

      Please return if something is unclear,
      best regards, Rickard

      Reply
  4. Andrew says:
    July 25, 2013 at 20:47

    I am having a problem where a vmotion completes successfully but the VM drop 4-6 pings instead of losing 1 or no pings. I have been troubleshooting this issue for several days and found your article helpful in explaining things, but I still have not solved the problem. I have the notify switches setting set on all hosts. I tried running wireshark on a vm that successfully vmotions with 1 or no ping drops (and vmotion that drops 4-6 pings) and I don’t see the rarp packets in either trace. All the ESXi logs seem to show the vMotion is successful, but for some reason the network switchover is slow. Any ideas?

    Reply
    1. Chandan says:
      July 17, 2020 at 13:01

      Andrew, even I’m facing problem similar to what you’ve mentioned. Were you able to find the root cause?

      Reply
  5. Mario says:
    August 23, 2013 at 15:33

    Great explanation Rickard!

    Thx!

    Reply
  6. Pablo says:
    March 12, 2015 at 23:22

    I wonder why not to send a gratuitous ARP instead: much more supported, not obsolete, and makes the same effect.

    Anyone knows?

    Reply
    1. Rickard Nobel says:
      March 12, 2015 at 23:28

      Hello Pablo,

      the main reason in my opinion is that it does not matter WHAT you send, the only thing important is that SOMETHING is sent with the hardware address of the VM as source MAC in the ethernet frame.

      The gratuitous ARP is not really needed since there is no change in the relation between IP and MAC, and it might possible disturb more machines on the network since everyone will process ARP (ethertype 0806), but rarely RARP (8035).

      Regards, Rickard

      Reply
  7. Joseph Green says:
    May 3, 2015 at 19:56

    Great writing. I will read more. Subscribing.

    Reply
  8. Christopher Sullivan says:
    July 15, 2015 at 15:31

    Great explanation, thanks!

    Reply
  9. hadi says:
    April 19, 2016 at 15:02

    please any answer for this comment

    I am having a problem where a vmotion completes successfully but the VM drop 4-6 pings instead of losing 1 or no pings. I have been troubleshooting this issue for several days and found your article helpful in explaining things, but I still have not solved the problem. I have the notify switches setting set on all hosts. I tried running wireshark on a vm that successfully vmotions with 1 or no ping drops (and vmotion that drops 4-6 pings) and I donโ€™t see the rarp packets in either trace. All the ESXi logs seem to show the vMotion is successful, but for some reason the network switchover is slow. Any ideas?

    Reply
    1. NB says:
      November 7, 2019 at 09:32

      Great article, though also experiencing this issue. Hope you can explain.

      I am having a problem where a vmotion completes successfully but the VM drop 4-6 pings instead of losing 1 or no pings. I have been troubleshooting this issue for several days and found your article helpful in explaining things, but I still have not solved the problem. I have the notify switches setting set on all hosts. I tried running wireshark on a vm that successfully vmotions with 1 or no ping drops (and vmotion that drops 4-6 pings) and I donโ€™t see the rarp packets in either trace. All the ESXi logs seem to show the vMotion is successful, but for some reason the network switchover is slow. Any ideas?

      Reply
      1. Rickard Nobel says:
        November 7, 2019 at 10:31

        Thanks for your comment!

        The RARP packets themselves will actually not be seen inside the VM who “sends” them, as it is the VMkernel that does this on its own, pretending to be the VM. If you place the Wireshark on some other port on the same physical VLAN you should be able to see them.

        As for the general questions, it is unfortunately very difficult to give any advice due to the many factors of the whole configuration unknown.

        Best regards, Rickard

        Reply
  10. BDonlinemart says:
    September 24, 2016 at 19:46

    This is a best document i have found related to topic, Someone refer me this link and found my problem solved easily.

    Reply
  11. Konrad Pizzuto says:
    November 19, 2018 at 15:48

    Good article, thank you.

    Reply
  12. Pingback: ESXi 6.7 Build 16075168 |VMware Communities | For servers
  13. Jacques MARENGO says:
    July 27, 2022 at 12:39

    Very cool article indeed.

    But back to 2022 with DVS 6.6 i’m still not able to say to Network team if my DVS is sending DVS wide, ESXi wide or VM wide arp table ๐Ÿ™‚

    I still have RAPR storm broadcast and don’t really know why (+19000 RARP request in 5 minutes)

    Any technical documentation on DVS 6.6 and RARP to advise?

    thanks ๐Ÿ™‚

    Reply
  14. Eric Pulvino says:
    August 4, 2022 at 14:55

    The issue is most modern network switches just ignore RARP entirely. So they don’t see the signaling at all that the MAC has moved to a new port.

    The other issue is that many networks are delivered on top of L3 EVPN Fabrics, which send type2 mac/ip routes to update other switches and identify where a MAC and IP address live on the network. GARPs will update both the MAC/IP and the port and allow the switch to send a new type2 mac/ip evpn route to all the other switches in the fabric to inform them of where the VM now lives.

    Reply
    1. Rickard Nobel says:
      January 13, 2024 at 08:19

      Thanks for your comment, Eric. A late reply:

      “The issue is most modern network switches just ignore RARP entirely. So they donโ€™t see the signaling at all that the MAC has moved to a new port.”

      I would argue that no device, either a switch or an operating system, supports RARP. However, that is not really necessary. The only important part is that this frame is passed on through the network, like any other Ethernet broadcast frame, and that the physical switches, automatically, as with any other frame regardless of the protocol, study the source MAC address field.

      Best regards, Rickard

      Reply
  15. Karl Brigl says:
    February 6, 2024 at 12:10

    Hi Rickard,
    I have the problem, that after a failover the gratuitous ARP is not delivered outside the vSphere cluster and so the clusterservice isn’t reachable till the arp cache of the network components is cleared. Is the ‘notifiy switches’ option the resolution or is there another option to configure?
    Thanx Karl

    Reply
    1. Rickard Nobel says:
      February 7, 2024 at 07:03

      Hello Carl,
      And thanks for your question. I don’t know what specific type of cluster service you are using, but you mention ARP and the ARP caches at some devices. If that being the case, I would say that the “Notify Switches” in not the solution.

      ARP is the relationship between Layer 3 (IPv4) and Layer 2 (the MAC address). Issues with ARP caches = the MAC address has changed, and a flush of the cache is required. A virtual machine being vMotioned or restarted at another host will remain the IP address AND the MAC address, so no flushing of ARP caches is required.

      The RARP protocol used with the “Notify Switches” feature is only used for an indirect purpose: to inform the switches of the relationship between Layer 2 (the VM MAC address) and Layer 1 (the physical interface in the physical network). This is due to while the VM MAC address has not changed, the location in the network is now different.

      Best regards, Rickard

      Reply
  16. Jithesh says:
    December 15, 2024 at 10:50

    Well explained, Thank you , 12 year old post ! still useful ๐Ÿ™‚

    Reply
    1. Rickard Nobel says:
      March 31, 2025 at 21:56

      Thanks Jithesh, I am glad to hear that. ๐Ÿ™‚

      Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Recent Posts

  • The ARP Probe frame
  • The Active Directory VM Generation-ID, part 4
  • The Active Directory VM Generation-ID, part 3
  • The Active Directory VM Generation-ID, part 2
  • The Active Directory VM Generation-ID, part 1
  • AD-joined appliances cannot use Kerberos AES.
  • AD Trust: The other domain supports Kerberos AES – explained.

Contact:

ยฉ2021 RICKARD NOBEL AB