RICKARD NOBEL AB

RICKARD NOBEL AB

Specialists in IT infrastructure services

Menu
  • About
  • Windows
  • Networking
  • VMware
  • Storage
Menu

Troubleshooting ESXi Jumbo Frames

Posted on November 10, 2011September 29, 2014 by Rickard Nobel

How to troubleshoot Jumbo Frames with VMKping on ESXi hosts or Ping on Windows servers and avoid common mistakes. Learn the correct syntax for VMKping or the ping tests will be useless.

“Jumbo frames” is the ability of using larger Ethernet frames than the default of 1518 bytes. As explained here this could increase the total throughput of the network and also decrease load on both hosts and storage. In this blog post we will see how to verify that Jumbo Frames has been correctly configured and actually is working.

When configuring Jumbo Frames we typically define the new MTU (Maximum Transmission Unit) which really is the payload inside an Ethernet Frame. For ordinary Ethernet the maximum frame size is 1518 byte, where 18 of these are used by Ethernet for header and checksum, which leaves 1500 byte to be carried inside the frame.

Since Jumbo Frames is not really a defined standard there is no actual rule on what MTU size to use, but a very common number is 9000, expanding the payload six times. This also means that the Ethernet frame size is 9018 byte, since MTU defines what could be carried inside the frame.

In VMware ESXi 5.0 there is a lot less work to configure Jumbo Frames than in earlier releases, since we have a GUI interface to change the MTU for both vSwitches and the VMkernel ports. Assuming that this has been done we now want to verify that it is actually working. The best way is to use the ESXi Shell, either directly through the ESXi console or with SSH.

We could now use the vmkping command (vmkernel ping) to try the larger frame sizes. There are however some common mistakes that should be avoided.

With vmkping we could use the -s option to set the ICMP payload size. The first mistake is to set this to 9000 and then try to ping for example the iSCSI SAN target. If this succeeds, as above, we might think Jumbo Frames has been confirmed end to end, but we have really not.

Vmkernel vmkping Jumbo Frames
Let us now try with a payload of 20000 bytes – and it works too! Perhaps really giant jumbo frames on this network? This might seems strange, but what is happening is ordinary IP level fragmentation. When IP becomes aware that it has a packet too large to fit into a frame it will be fragmented and all kind of sizes will seem to work fine. If we actually want to force IP to not “help” us with our Jumbo frames test we must attach the -d flag to vmkping, which forbids IP fragmentation. On a Windows server we would use -f.

When IP fragmentation is removed we can see (above) that our ping is failing. This might imply that Jumbo Frames is not working, but it could very well do – since there is another common mistake with the vmkping command.

The -s flag for size is not the total packet size, but the ICMP payload, i.e. the number of bytes after the ICMP header. ICMP itself consumes 8 byte and is on top of IP which has a 20 byte header.

Since the MTU is 9000, but we consume 8+20 for ICMP and IP, we would have to set the payload size to 28 less than 9000, that is 8972 bytes.

When this works then we have finally proven that the Jumbo frame with 9000 MTU is working from our host through the network to the target.

So for ESXi hosts use: vmkping -s 8972 -d IP

On Windows Server we could use ordinary Ping in the command prompt with the options -l (lowercase L) for payload length and -f for Do not Fragment: ping -l 8972 -f IP

Remember to test this from all hosts / servers that should be able to access a storage device with jumbo frames. This verifies both host configuration, but also physical switches through the network.

17 thoughts on “Troubleshooting ESXi Jumbo Frames”

  1. Anders Olsson says:
    February 5, 2013 at 14:11

    Thank you for a great guide and good explainations. Helped me figure out a missed MTU 9000 on a vmk port.

    Reply
    1. Rickard Nobel says:
      February 5, 2013 at 14:13

      Nice to hear that it was helpful Anders. Thank you for your comment,
      regards Rickard

      Reply
  2. Mohammed M says:
    March 3, 2013 at 14:24

    Thanks for a clear and lucid explanation of the overhead bytes Rickard – very useful!

    Reply
    1. Rickard Nobel says:
      March 3, 2013 at 14:27

      Glad to be able to help, thanks for commenting!

      Reply
  3. Rob says:
    April 11, 2013 at 18:04

    Great article, very well written! This helped me with a problem I was working on today.

    Reply
    1. Rickard Nobel says:
      April 11, 2013 at 18:08

      Nice to hear that Rob, thanks for your comment.

      Regards, Rickard

      Reply
  4. Pingback: Обзор блогов от 15/04/13 | vMind.ru
  5. Josh says:
    July 20, 2013 at 17:33

    Hi there, I’m having trouble getting this working on my host.

    I can do the test as you can here but my Switch cannot see ANY jumbo frames coming from that interface, which would suggest it is still fragmenting the packets for some reason.

    Got a thread here if you could help: http://communities.vmware.com/message/2266519#2266519

    Reply
  6. Sebastien Ollier says:
    November 4, 2015 at 21:40

    Hi Rickard, Thank’s for your article.

    Reply
  7. Stefand says:
    April 14, 2016 at 09:51

    Thank you for this great post! I was struggling with my ESX hosts and Dell Force10 switches. When I configure an MTU of 9000 on the Dell Force10 Switch Jumboframes was not working.

    When I configure a MTU of 9018, everything is working!

    Keep up the good work!

    Reply
    1. Rickard Nobel says:
      April 14, 2016 at 10:07

      Hello Stefan,

      and thank you for your comment! Glad that you were able to fix your problem.

      It is interesting to note that the switch setting of MTU=9000 should really work IF properly handled by the switch. Since the MTU is really only the payload inside the ethernet frame the “outer” 18 bytes should be implied. It seems however that your switches misunderstood the MTU value and used it like “maxiumum frame size”, which is incorrect.

      Please note that if you are using VLAN tagging then the frame size will be 4 bytes extra. It will not affect the MTU, but the total frame size which could be 9022 bytes in that case.

      Best regards,
      Rickard

      Reply
  8. James M says:
    May 15, 2016 at 18:07

    A great post. Many thanks.

    Reply
  9. Vishal Thombare says:
    December 27, 2016 at 12:19

    Thanks , This is very helpful.

    Reply
  10. Pete says:
    September 14, 2017 at 12:13

    Hi Rick – This post was a great help whilst I was tracking down a HP VSA \ ESXi jumbo frames issue and allowed me to track down whether Jumbo frames had been on or off and confirm the state of play – Many thanks

    Reply
  11. basavaraj R Navalgund says:
    April 22, 2018 at 10:33

    Hi Ricard,

    Thank you for this great post, which definitely helps a vm admin or architect.
    I have a small query,

    Are there any vmware proprietorial tools /command to very the same. OR still this methods is the only easy method to verify the JF flow,

    Regards
    Raj_Navalgund
    Bangalore

    Reply
  12. linksys e1200 says:
    July 22, 2019 at 10:41

    Thanks, I have solved my jumbo frame issue thanks to your article.

    Reply
  13. Lijo George says:
    March 4, 2020 at 06:48

    Thanks for the explanation. However, when I try payload size 8972 with vmkping, it fails, and when I try 8950 and any lower value, it works. Does that mean Jumbo frame is really in action in storage communication? I’m ok to have frame sizes slightly lower than 8972, but want make sure it is not as low as 1518. Kindly advise.

    Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Recent Posts

  • Verify NTP connectivity in Windows
  • The Ethertype value, part 1
  • Password strength part 1, the mathematical basics
  • MS16-072 breaks Group Policy
  • ESXi virtual machine network statistics
  • Determine the Zeroed status of Thick Lazy disk
  • Eager thick vs Lazy thick disk performance

Contact

Categories

  • Networking
  • Storage
  • VMware
  • Windows

Recent Comments

  • Rickard Nobel on VMXNET3 vs E1000E and E1000 – part 1
  • cees vos on VMXNET3 vs E1000E and E1000 – part 1
  • Filipi Souza on Storage performance: IOPS, latency and throughput
  • Backup vs RAID - Web Hosting on How RAID 5 actually works
  • Stephen on Password strength part 1, the mathematical basics
©2021 RICKARD NOBEL AB