VMWare shared storage optimization question (blocksizes)

Discussion in 'VMware, VirtualBox, Citrix' started by Rand__, Apr 22, 2018.

  1. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,343
    Likes Received:
    290
    Hi,

    been wondering for a while and saw other ppl have the same question but didn't find a definitive answer, so here it goes...

    When building a shared storage for a ESX box (or cluster) - what is the most appropriate measurement to take in running synthetic benchmarks?

    O/c one must differentiate between infrastructure activity (create/move vms) and application activity (Exchange/Database VM).

    The latter will usually depend on the application so its difficult to specify an universal value.

    Therefore i'd like to restrict this to infrastructure, so
    -A - VMotion
    -B - Create a new VM (thick provision)
    -C - maybe regular OS activity

    Secondary aspect is o/c the number of concurrent threads (do I need low QD optimized drives or deep QD) but thats fairly individual too so I'd like to look at low QD.


    I always looked for 4K random writes but nowadays I am not sure thats it.
    I ran some tests recently on a Compuverde cluster and they have a nice Gui feature where you can (roughly) see the block size being used.
    I have not checked B or C explicitly but A was in between <128K (but not <64K) segment so it might be 64K.

    Does anyone have more specific info?
     
    #1
  2. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,323
    Likes Received:
    302
    #2
  3. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,343
    Likes Received:
    290
    Well that describes the storage block size, so that might be applicable for B and C.
    1MB standard (user configurable at datastore creation time)

    The other question is what is the transfer chunk size on a vmotion?
    Is that read block wise (say 1MB) and then split into TCP frames (1500 or 9k) and then spewed out at whatever comes in? Or aggregated at NIC/OS level and written in 1MB chunks? (only to be split into smaller chunks at physical level again)
     
    #3
  4. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,323
    Likes Received:
    302
    That was true for vmfs <5, now it's 1mb only
    If I remeber correctly files/data (the 1mb block) is split in smaller datagrams (tcp, up to 64kb) and then again at ethernet level (layer 2 in osi, 1500~9kb). On the other side the os puts the tcp datagrams back together in the right order and writes it (the 1mb block) to the filesystem.
     
    #4
  5. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,343
    Likes Received:
    290
    so read would be 64K blocks and Write 1MB chunks?

    Then its weird I saw 64KB (most liekly) block writes on Compuverde
     
    #5
  6. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,323
    Likes Received:
    302
    No, the os (esxi) reads a 1mb block from the block device which could be a ssd, a hdd or a lun into ram.

    In the tcp layer the 1mb block can be split into 64kb segments and so on:
    [​IMG]
     
    #6
  7. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,343
    Likes Received:
    290
    So everything is read/written at 1MB blocks? Hm it did not look that way, weird
     
    #7
  8. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,323
    Likes Received:
    302
    Yes and no :D

    Esxi read and writes in 1mb parts. IF you have a lun or nfs server it will depend on the underlying storage and how it's implemented and so on.

    Lets say you have a raid 10/zfs mirrors with 8 ssds and a stripe size of 128kb. When esxi writes a block the storage server will split the data in 128kb stripes and distribute it over the mirrors (each mirror has to write two stripes a la 128kb).

    That's probably what you have seen in the compuverde cluster.

    :)D)
     
    #8
  9. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,343
    Likes Received:
    290
    Hm that might be.
    But then the display would be kind of useless as it would always show the internal decided/configured blocksize.

    Will run some tests if a 1MB ATTO or fio run displays similar speeds as a vmotion :)

    Thanks
     
    #9
Similar Threads: VMWare shared
Forum Title Date
VMware, VirtualBox, Citrix Shared storage options for VMware ESXi Aug 15, 2016
VMware, VirtualBox, Citrix Migrate Xenserver to VMware - Good idea? Aug 2, 2018
VMware, VirtualBox, Citrix VMWare vSAN, quad M.2 NVME + 1 cache vs SATA SSDs? Jul 5, 2018
VMware, VirtualBox, Citrix VMware Guest RAID pass through question May 1, 2018
VMware, VirtualBox, Citrix VMware 6.7 Issue with intel x540-t2 SR-IOV Apr 30, 2018

Share This Page