VMWare shared storage optimization question (blocksizes)

Discussion in 'VMware, VirtualBox, Citrix' started by Rand__, Apr 22, 2018.

  1. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,484
    Likes Received:
    325
    Hi,

    been wondering for a while and saw other ppl have the same question but didn't find a definitive answer, so here it goes...

    When building a shared storage for a ESX box (or cluster) - what is the most appropriate measurement to take in running synthetic benchmarks?

    O/c one must differentiate between infrastructure activity (create/move vms) and application activity (Exchange/Database VM).

    The latter will usually depend on the application so its difficult to specify an universal value.

    Therefore i'd like to restrict this to infrastructure, so
    -A - VMotion
    -B - Create a new VM (thick provision)
    -C - maybe regular OS activity

    Secondary aspect is o/c the number of concurrent threads (do I need low QD optimized drives or deep QD) but thats fairly individual too so I'd like to look at low QD.


    I always looked for 4K random writes but nowadays I am not sure thats it.
    I ran some tests recently on a Compuverde cluster and they have a nice Gui feature where you can (roughly) see the block size being used.
    I have not checked B or C explicitly but A was in between <128K (but not <64K) segment so it might be 64K.

    Does anyone have more specific info?
     
    #1
  2. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,384
    Likes Received:
    317
    #2
  3. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,484
    Likes Received:
    325
    Well that describes the storage block size, so that might be applicable for B and C.
    1MB standard (user configurable at datastore creation time)

    The other question is what is the transfer chunk size on a vmotion?
    Is that read block wise (say 1MB) and then split into TCP frames (1500 or 9k) and then spewed out at whatever comes in? Or aggregated at NIC/OS level and written in 1MB chunks? (only to be split into smaller chunks at physical level again)
     
    #3
  4. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,384
    Likes Received:
    317
    That was true for vmfs <5, now it's 1mb only
    If I remeber correctly files/data (the 1mb block) is split in smaller datagrams (tcp, up to 64kb) and then again at ethernet level (layer 2 in osi, 1500~9kb). On the other side the os puts the tcp datagrams back together in the right order and writes it (the 1mb block) to the filesystem.
     
    #4
  5. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,484
    Likes Received:
    325
    so read would be 64K blocks and Write 1MB chunks?

    Then its weird I saw 64KB (most liekly) block writes on Compuverde
     
    #5
  6. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,384
    Likes Received:
    317
    No, the os (esxi) reads a 1mb block from the block device which could be a ssd, a hdd or a lun into ram.

    In the tcp layer the 1mb block can be split into 64kb segments and so on:
    [​IMG]
     
    #6
  7. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,484
    Likes Received:
    325
    So everything is read/written at 1MB blocks? Hm it did not look that way, weird
     
    #7
  8. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,384
    Likes Received:
    317
    Yes and no :D

    Esxi read and writes in 1mb parts. IF you have a lun or nfs server it will depend on the underlying storage and how it's implemented and so on.

    Lets say you have a raid 10/zfs mirrors with 8 ssds and a stripe size of 128kb. When esxi writes a block the storage server will split the data in 128kb stripes and distribute it over the mirrors (each mirror has to write two stripes a la 128kb).

    That's probably what you have seen in the compuverde cluster.

    :)D)
     
    #8
  9. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,484
    Likes Received:
    325
    Hm that might be.
    But then the display would be kind of useless as it would always show the internal decided/configured blocksize.

    Will run some tests if a 1MB ATTO or fio run displays similar speeds as a vmotion :)

    Thanks
     
    #9
Similar Threads: VMWare shared
Forum Title Date
VMware, VirtualBox, Citrix Shared storage options for VMware ESXi Aug 15, 2016
VMware, VirtualBox, Citrix VMware vSphere Hypervisor (ESXi) and vCenter Server 6.7U1 released Tuesday at 2:51 PM
VMware, VirtualBox, Citrix Getting Started with VMWare, Replication, Veeam, etc Sep 28, 2018
VMware, VirtualBox, Citrix Migrate Xenserver to VMware - Good idea? Aug 2, 2018
VMware, VirtualBox, Citrix VMWare vSAN, quad M.2 NVME + 1 cache vs SATA SSDs? Jul 5, 2018

Share This Page