Eager thick vs Lazy thick disk performance

By | August 26, 2015

Eager-vs-Lazy-Disk

The VMware Thick Eager Zeroed Disk vs the Lazy Zeroed Thick disk in write performance.

What is the potential write performance difference between the VMware virtual disks: Thick Lazy Zeroed, Thick Eager Zeroed and Thin provisioned types? This has been discussed for many years and there are many opinions regarding this, both in terms of test vs real life write behavior as well as test methods. There is also important factors as storage effiency, migration times and similar to this, however I will in this article try to make the potential “first write” impact more easy to evaluate.

Before the virtual machine guest operating system can actually use a virtual disk some preparations has to be accomplished by the ESXi host. The main tasks that has to be done for each writable part of a virtual disk is that the space has to be allocated on the datastore and the specific disk sectors on the storage array has to be safely cleared (“zeroed”) from any previous content.

In short, this is done in the following way:

Thin: Allocate and zero on first write
Thick Lazy: Allocate in advance and zero on first write
Thick Eager: Allocate and zero in advance

There are some published performance tests between these three disk types often using the standard tool IOmeter. There is however a potential flaw to these tests, caused by the fact that before IOmeter starts the actual test it will create a file (iobw.tst) and write data to each part of that file – which at the same time causes ESXi to zero out these blocks on the storage array. This means that it is impossible to use IOmeter output data to spot any write performance differences between the three VMware virtual disk types, since the potential difference in write performance will already be nullified when the IOmeter test actually begins.

When the difference will only be in the very first write from the Virtual Machine across the virtual disk sectors a way to simulate this is to force a massive amount of writes over the whole disk area and note the time differences. This is of course not how most applications work in the sense that it is uncommon to do all writes in one continuous stream and instead the “first-writes” with ESXi zeroing is likely to be spread over a longer period of time, but sooner of later each sector that is used by the guest operating system has to be zeroed.

Eager-vs-Lazy-Disk

A way to simulate large amounts of writes could be done from using the standard Windows format tool which, despite some popular belief, actually erases the whole disk area if selecting a “full” / non-quick format. In real life there is not much specific interest how fast a partition format is done in itself, however in this test the format tool is used just to create a massive amount of “first-writes”.

Eager-vs-Lazy-Disk

This test case uses a VM with Windows 2012 R2 which was given three new virtual hard disks of 40 GB each, where there was one Eager, one Lazy and one Thin. Each disk was then formatted with NTFS, default allocation unit, no compression, but with the non-quick option.

Eager-vs-Lazy-Disk

One first observation was in ESXTOP while looking at the ratio between the writes that the virtual machine actually commits compared to how many writes are being sent from ESXi to the LUN.

Above we can see ESXTOP while doing a full format of an Eager Zeroed Thick disk. The important point here is that the numbers are very close. The writes being done at the LUN are only the writes the VM wants to make, i.e. there is no ESXi introduced extra writes since the “zeroing” was done already in advance.

Eager-vs-Lazy-Disk

Above a Lazy Zeroed Thick Disk is being full formated from inside the VM.

What could be noticed is the amount of write IOs being sent from ESXi to the LUN is much higher than the number of write IOs coming from the virtual machines. This is the actual zeroing taking place “in real time” and will make the VM write performance lower than the Eager version while accessing new areas of the virtual disk for the first time.

Eager-vs-Lazy-Disk

The actual time results for doing a full format of a 40 GB virtual disk was:

Eager Zeroed Thick Disk: 537 seconds
Lazy Zeroed Thick Disk: 667 seconds
Thin Disk: 784 seconds

The Eager Zeroed Thick Disk was almost 25 % faster in first-write performance compared to the Lazy Zeroed.

The Eager Zeroed Thick Disk was almost 45 % faster in first-write performance compared to the Thin Disk.

This is obvious in doing a full format which forces the VM to write at all sectors. In a real environment the “first-writes” will naturally be spread over a longer period of time, but sooner or later the Zeroing hit will take place for each part of the disk and might or might not be noticeable to the user. For a typical virtual machine that does the majority of “first-writes” at OS installation this is likely to be of lesser interest, but for VMs with databases, logfiles or other write intensive applications it is possible to result in a higher impact.

Eager-vs-Lazy-Disk

One important thing to notice is also that after the first-write is done the write performance is the same between all three disk types. This could be proven by doing several full formats of the same VMDK disk. On the first format the zeroing will be done (for the Lazy and Thin types), but when the whole disk is zeroed the write performance is identical.

Above a 5 GB VMDK disk was formated three times. The Eager was the fastest on the first format since the zeroing is done before the disk is even visible to the VM and the Lazy and Thin was slower as expected on the initial format run.

However, after the zeroing from ESXi is done the write performance is identical, which is visible above where all three disk performs the same on format run 2 and 3 on the same disk.

The Eager Zeroed Thick Disk will be faster on first write IO on each new part of the disk where Lazy and Thin is slower, however after the disk blocks are zeroed the write performance is the same.

8 thoughts on “Eager thick vs Lazy thick disk performance

  1. Filibuster

    Hi!
    Great article!

    You don’t mention where the test LUN’s are provisioned from. I guess there are big differences on how zeroing are handled on different modern storage arrays (often pointer based), and also how complete, and how good the different VAAI features are implemented by the different vendors.

    Reply
    1. Rickard Nobel Post author

      Hello,

      and thank you for your comment. You make a good point about the importance of the storage array, the system was a HP 3PAR StorServe with the specific LUN backed only by SSD drives. The 3PAR has all the VAAI capabilities which is very good of course, however I am not sure how much they do make a difference in this case.

      The VAAI write-same is great for creating the Eager Zeroed Thick type, the 40 GB disk used here took only about 13 seconds to create – likely a combination of the VAAI write-same together with the 3PAR zero-detect function. This makes it very quick to create the VMDK itself but does not really change the VM access after the VMDK is presented to the VM which was studied here.

      The 3PAR has hardware detection of “zero writes” to save space, but it was impressive to see that this did not make the access any slower when the “zeroed” block was actually used by the VM.

      Another VAAI function that might impact is the ATS, which makes less (or none) SCSI reservation needed when expanding the Thin disk type. My guess is that the ATS capability did help the Thin being faster in the test (and in production also less impact on other VMs on the same LUN), but the thin type was still the slowest of the three.

      Best regards, Rickard

      Reply
  2. raihan

    can we know where the log was locate for activity on changes from eg: from lazy to eager zero.
    I’m try to find from esxi vmkernel and also *.log, at the moment did not find.

    Reply
    1. Rickard Nobel Post author

      Hello,

      I have not been looking specifically for the log entries together with disk type changes, so I have no answer for you unfortunately.

      Regards, Rickard

      Reply
  3. notStanley

    Found this article from a link in the Veeam Forums Digest. Just to be a little contrary, we need to acknowledge that the minor improvement of Thick-Eager at the beginning of a vmdk life cycle is only important for very specific high performance machines. Do not let anyone think this means they should never use Thin! If all our non-critical machines were Thick we could never afford all the storage to provision them after making the volumes large enough for the OS and Applications to see “recommended” free space. Such as customers who insist they must have a 2TB data volume, even though three years later they have still only used 450GB :{

    Reply
    1. Nick

      I still prefer to let the SAN handle the over-provisioning so would only use thin-provisioning on a VM for DAS storage personally, I’d certainly avoid combining both.
      A good article here though and probably as good a test as is feasible without setting up 3 identical system with apps/DBs and complex test scripts simulating user activity. Personally I always provision thick/lazy, mostly as I’m not convinced it makes much real-world difference (and this article doesn’t change my mind on that) and I generally can’t be bothered waiting for multi-TB VMDK creation using thick/eager

      Reply
  4. Joost

    Very important article to understand the performance differences between them, thank you!

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *