RICKARD NOBEL AB

RICKARD NOBEL AB

Specialists in IT infrastructure services

Menu
  • About
  • Windows
  • Networking
  • VMware
  • Storage
Menu

The vSwitch “Notify Switches” setting

Posted on July 14, 2012September 29, 2014 by Rickard Nobel

On the vSphere virtual switch we have the possibility to enable or disable the option of “Notify Switches”. In this post we shall see what this setting actually does and how it works, as well as discussing if the RARP protocol is important.

notify-switches

To see the current setting or to change it, select Properties on a vSwitch, then Edit and then the NIC Teaming tab.

When having the value set to Yes it will basically give ESXi the permission to sometimes send faked frames on behalf of the Virtual Machines running on the host.

The goal for sending these frames is to make sure the physical switches in the network learns the location of the Virtual Machines. A physical switch does this learning by observing each incoming frame and make a note of the field called Source MAC Address. Based on that information the switches build tables with mappings between MAC addresses and the switch port where this address could be found.

There are at least three different occasions where these messages are sent by ESXi:

1. When a Virtual Machine is powered on the ESXi host will make sure the network are aware of this new VM and its MAC address as well as its physical location (switch port).

2. When a Virtual Machine is moved by vMotion to another host it will be very crucial to rapidly notify the physical network of the new placement of the virtual server.

notify-switches

3. If a physical NIC on the host loses the connection, but we had at least two vmnics as uplinks on the vSwitch all traffic from the VMs will be “moved” to the remaining interface. Also in this situation the ESXi host must very quickly inform the network of the new correct uplink to reach the VM.

The protocol used is RARP – Reverse Address Resolution Protocol. Sometimes in texts about vSphere it is implied that RARP itself is important and actually does something in this process. However, RARP is a very old and obsolete protocol which almost no operating systems or devices has support for today.

So why send frames that almost nobody could read? The reason is that it is only important that all physical switches sees the Source MAC address and learn the new location and in reality the payload (in this case RARP) could be almost anything.

notify-switches

Here we see a virtual machine with a certain MAC address. This VM is being migrated with vMotion and with Wireshark we shall see which RARP frames being sent.

notify-switches

In this packet trace we can notice that the ESXi host is sending RARP frames with the VM as source MAC and also that it wants to be very certain that every switch has been reached by this new information – and actually sends the frame ten times. First several times close together and then more spread in time, with the last frame almost one minute after the first information frame.

notify-switches

How could we be sure that all switches on the physical network does see these frames? This is because of the destination is the special broadcast address FF-FF-FF-FF-FF-FF. The broadcast address forces all switches to forward the frame on all ports and makes the whole broadcast domain aware of the new location.

If the Notify Switches settings for some reason is set to No then the behavior is changed and the host will be silent when actions like VM power-on, vMotion and vmnic link failure take place. It will likely still work, but a much higher risk of packet loss in the above situations.

In summary, the Notify Switches function is very useful and should be set to Yes to allow the ESXi host acting as the virtual machines and sends faked frames to make sure all physical switches MAC-to-port mapping tables are quickly updated.

18 thoughts on “The vSwitch “Notify Switches” setting”

  1. Gopikeerthy says:
    July 15, 2012 at 19:10

    Hi,

    Really great information from you, now I got the very in depth theory of the RARP in ESXi. Thanks for your effort.

    / Gopi

    Reply
  2. Mohan says:
    December 23, 2012 at 14:44

    We are likely facing the situation explined in – http://kb.vmware.com/kb/1556.

    After seeing your excellent explanation of this functionality, I dont want to loose this, so is there a way that I can still leave this setting to Yes but some how instruct the host to use the bogus MAC address (in its RARP) for each NLB guest?

    If I register static arp entries on the hosts for each NLB guest, will this force the hosts to use such bogus MAC in RARP? Any help would be much appreciated, thanks in advance.

    Reply
    1. Rickard Nobel says:
      December 23, 2012 at 15:28

      Hello Mohan,
      and thank you for your reply.

      From my understanding of Microsoft NLB Clusters I think the key point is that the NLB machines (physical or virtual) depend on that switches never learn their true position and always floods every incoming frame to all destination ports, making sure this hits all the NLB machines.

      Because of this it should be recommended to keep the NLB machines into their own VLAN to limit the frame flooding. If so you should be able to disable the “VMware Notify Switches” setting on these portgroups only, but keep the rest of your VM portgroups with the Notify setting enabled.

      Reply
  3. wildwill says:
    April 28, 2013 at 05:30

    This is great information for me to understand why vMotion sends RARP packets after migration is over. But one question is that what if the server is connected with a Layer 3 switch/router directly ?

    If RARP is not processed by switch/router (which is commonly seen now), this means that MAC:Port is updated by RARP packet but IP:MAC not. This could make influence on data traffic.

    Reply
    1. Rickard Nobel says:
      April 28, 2013 at 08:23

      Thank you for your comment wildwill.

      Does your question concerns having one ESXi host directly connected to a routing device with no layer two switching between and another ESXi on the remote side of the router?

      For vMotion to actually work you would need to have a common layer two infrastructure, both for the vmkernel adapters with vMotion tag and for all Virtual Machine networks. That would mean that connecting two ESXi host to opposite sides of a router would not work with vMotion. (It could be done with different tunneling techniques, but not out of the box.)

      As for the relation IP to MAC, typically done with classic ARP resolution, the Virtual Machine itself is not in any way changed in the vMotion process. This means the virtual network card will keep the same MAC address and the TCP/IP stack inside the guest operating system will have the same IP address.

      The effect of that nothing changes inside the guest is that in a vMotion transfer there is also no need to change or update any existing ARP entries on any external devices.

      Please return if something is unclear,
      best regards, Rickard

      Reply
  4. Andrew says:
    July 25, 2013 at 20:47

    I am having a problem where a vmotion completes successfully but the VM drop 4-6 pings instead of losing 1 or no pings. I have been troubleshooting this issue for several days and found your article helpful in explaining things, but I still have not solved the problem. I have the notify switches setting set on all hosts. I tried running wireshark on a vm that successfully vmotions with 1 or no ping drops (and vmotion that drops 4-6 pings) and I don’t see the rarp packets in either trace. All the ESXi logs seem to show the vMotion is successful, but for some reason the network switchover is slow. Any ideas?

    Reply
    1. Chandan says:
      July 17, 2020 at 13:01

      Andrew, even I’m facing problem similar to what you’ve mentioned. Were you able to find the root cause?

      Reply
  5. Mario says:
    August 23, 2013 at 15:33

    Great explanation Rickard!

    Thx!

    Reply
  6. Pablo says:
    March 12, 2015 at 23:22

    I wonder why not to send a gratuitous ARP instead: much more supported, not obsolete, and makes the same effect.

    Anyone knows?

    Reply
    1. Rickard Nobel says:
      March 12, 2015 at 23:28

      Hello Pablo,

      the main reason in my opinion is that it does not matter WHAT you send, the only thing important is that SOMETHING is sent with the hardware address of the VM as source MAC in the ethernet frame.

      The gratuitous ARP is not really needed since there is no change in the relation between IP and MAC, and it might possible disturb more machines on the network since everyone will process ARP (ethertype 0806), but rarely RARP (8035).

      Regards, Rickard

      Reply
  7. Joseph Green says:
    May 3, 2015 at 19:56

    Great writing. I will read more. Subscribing.

    Reply
  8. Christopher Sullivan says:
    July 15, 2015 at 15:31

    Great explanation, thanks!

    Reply
  9. hadi says:
    April 19, 2016 at 15:02

    please any answer for this comment

    I am having a problem where a vmotion completes successfully but the VM drop 4-6 pings instead of losing 1 or no pings. I have been troubleshooting this issue for several days and found your article helpful in explaining things, but I still have not solved the problem. I have the notify switches setting set on all hosts. I tried running wireshark on a vm that successfully vmotions with 1 or no ping drops (and vmotion that drops 4-6 pings) and I don’t see the rarp packets in either trace. All the ESXi logs seem to show the vMotion is successful, but for some reason the network switchover is slow. Any ideas?

    Reply
    1. NB says:
      November 7, 2019 at 09:32

      Great article, though also experiencing this issue. Hope you can explain.

      I am having a problem where a vmotion completes successfully but the VM drop 4-6 pings instead of losing 1 or no pings. I have been troubleshooting this issue for several days and found your article helpful in explaining things, but I still have not solved the problem. I have the notify switches setting set on all hosts. I tried running wireshark on a vm that successfully vmotions with 1 or no ping drops (and vmotion that drops 4-6 pings) and I don’t see the rarp packets in either trace. All the ESXi logs seem to show the vMotion is successful, but for some reason the network switchover is slow. Any ideas?

      Reply
      1. Rickard Nobel says:
        November 7, 2019 at 10:31

        Thanks for your comment!

        The RARP packets themselves will actually not be seen inside the VM who “sends” them, as it is the VMkernel that does this on its own, pretending to be the VM. If you place the Wireshark on some other port on the same physical VLAN you should be able to see them.

        As for the general questions, it is unfortunately very difficult to give any advice due to the many factors of the whole configuration unknown.

        Best regards, Rickard

        Reply
  10. BDonlinemart says:
    September 24, 2016 at 19:46

    This is a best document i have found related to topic, Someone refer me this link and found my problem solved easily.

    Reply
  11. Konrad Pizzuto says:
    November 19, 2018 at 15:48

    Good article, thank you.

    Reply
  12. Pingback: ESXi 6.7 Build 16075168 |VMware Communities | For servers

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Recent Posts

  • Verify NTP connectivity in Windows
  • The Ethertype value, part 1
  • Password strength part 1, the mathematical basics
  • MS16-072 breaks Group Policy
  • ESXi virtual machine network statistics
  • Determine the Zeroed status of Thick Lazy disk
  • Eager thick vs Lazy thick disk performance

Contact

Categories

  • Networking
  • Storage
  • VMware
  • Windows

Recent Comments

  • Rickard Nobel on VMXNET3 vs E1000E and E1000 – part 1
  • cees vos on VMXNET3 vs E1000E and E1000 – part 1
  • Filipi Souza on Storage performance: IOPS, latency and throughput
  • Backup vs RAID - Web Hosting on How RAID 5 actually works
  • Stephen on Password strength part 1, the mathematical basics
©2021 RICKARD NOBEL AB