ESX Slow NVMe with intel driver

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Just found this, thought it might interest ppl
VMware Knowledge Base



Low I/O performance using intel-nvme drivers 1.3.2.8-1OEM and 1.3.2.4-1OEM with block sizes larger than 4K (55693)
Symptoms
  • Significant performance degradation is observed when I/O block size is greater than 4K.
  • 4K block size IOPS are not affected.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Actually started and cause a near catastrophic datastore loss :O
Not necessarily the new drivers fault, but not entirely sure its not related...

I installed the new driver on one of my 3 hosts. Then I managed to set maintenance mode on via cli for one (wrong) host and since it didnt show in the gui i set again for the host I wanted to reboot.

When I noticed I tried cancelling the cli command, didnt work, closed putty.
The other box didnt go into maint but my VMs dropped all off...

So rebooted box 3 (with new driver) - but nothing came up again.
Rebooted box 1 to see whether that helped, but no go.

Rebooted box3 again after reinstalling old driver, still no difference.

At that point the whole estate was gone...
interestingly individual VMs showed 1 component present (box2) and one + witness missing (box 1/3).
Vsan health (physical disks) overall was fine so no clue....

Searching the web turned out no results ...

So in the end I managed to recover by explicitly setting maintenance mode again for box1/3 and then reboot them (twice each).

After everything was back to normal I started new
Have now rebooted box2 and am observing... there are some warnings in the manual regarding potential data store disappearance ...


edit: Forgot to mention that of course my backup VM was also on the vSan Datastore so no easy restore either :O
 
Last edited:

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
So totally forgot to take a current benchmark before going off, so not really a scientific result

nvme_1.2.1.15 on a single system
upload_2018-8-19_20-1-4.png

intel-nvme-vmd_1.4.0.1016
upload_2018-8-19_20-2-48.png