ESX Slow NVMe with intel driver

Discussion in 'VMware, VirtualBox, Citrix' started by Rand__, Jun 15, 2018.

  1. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,416
    Likes Received:
    302
    Just found this, thought it might interest ppl
    VMware Knowledge Base



    Low I/O performance using intel-nvme drivers 1.3.2.8-1OEM and 1.3.2.4-1OEM with block sizes larger than 4K (55693)
    Symptoms
    • Significant performance degradation is observed when I/O block size is greater than 4K.
    • 4K block size IOPS are not affected.
     
    #1
    Tha_14, svtkobra7 and Aestr like this.
  2. Aestr

    Aestr Active Member

    Joined:
    Oct 22, 2014
    Messages:
    654
    Likes Received:
    169
    Interesting. Thanks for the heads up!
     
    #2
  3. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,416
    Likes Received:
    302
  4. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,359
    Likes Received:
    1,296
    #4
  5. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,416
    Likes Received:
    302
    Actually started and cause a near catastrophic datastore loss :O
    Not necessarily the new drivers fault, but not entirely sure its not related...

    I installed the new driver on one of my 3 hosts. Then I managed to set maintenance mode on via cli for one (wrong) host and since it didnt show in the gui i set again for the host I wanted to reboot.

    When I noticed I tried cancelling the cli command, didnt work, closed putty.
    The other box didnt go into maint but my VMs dropped all off...

    So rebooted box 3 (with new driver) - but nothing came up again.
    Rebooted box 1 to see whether that helped, but no go.

    Rebooted box3 again after reinstalling old driver, still no difference.

    At that point the whole estate was gone...
    interestingly individual VMs showed 1 component present (box2) and one + witness missing (box 1/3).
    Vsan health (physical disks) overall was fine so no clue....

    Searching the web turned out no results ...

    So in the end I managed to recover by explicitly setting maintenance mode again for box1/3 and then reboot them (twice each).

    After everything was back to normal I started new
    Have now rebooted box2 and am observing... there are some warnings in the manual regarding potential data store disappearance ...


    edit: Forgot to mention that of course my backup VM was also on the vSan Datastore so no easy restore either :O
     
    #5
    Last edited: Aug 19, 2018
  6. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,416
    Likes Received:
    302
    So totally forgot to take a current benchmark before going off, so not really a scientific result

    nvme_1.2.1.15 on a single system
    upload_2018-8-19_20-1-4.png

    intel-nvme-vmd_1.4.0.1016
    upload_2018-8-19_20-2-48.png
     
    #6
Similar Threads: Slow NVMe
Forum Title Date
VMware, VirtualBox, Citrix Slow vmotion in ESX (nvme2nvme@56GbE) Feb 26, 2017
VMware, VirtualBox, Citrix Very slow esxi 6 or 6.5 response with X9SRH motherboard Mar 21, 2018
VMware, VirtualBox, Citrix Painfully slow Storage vMotion Migrations? Nov 15, 2017
VMware, VirtualBox, Citrix ESXI Backup with VEEAM concurrent tasks fast, single tasks slow Aug 16, 2017
VMware, VirtualBox, Citrix Esxi 6.5, x10sdv-4c-7tp4f, Samsung843 - slow performance? Mar 7, 2017

Share This Page