VMware vSAN Performance :-(

[Yves]

Thanks a lot
Fio - fio | Bluestop
Index of /files/fio/releases/archive/ (for 2.0.9)

- I currently use 2.09 on my linux box, if you could use that too that would be great as different versions are not always comparable

You're welcome. Ehm... found bluestop fio but the issue is the libaio engine which I don't seam to find for windows... :-S any trick on that?

 

[Rand__]

--ioengine=windowsaio



You also need --size

C:\Program Files\fio>fio --filename=test --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --ioengine=windowsaio --bs=4k --rwmixread=100 --iodepth=1 --numjobs=1 --runtime=60 --group_reporting --name=4kreadtest --size=8GB
fio: this platform does not support process shared mutexes, forcing use of threads. Use the 'thread' option to get rid of this warning.
4kreadtest: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=windowsaio, iodepth=1
fio-2.0.9
Starting 1 process
4kreadtest: Laying out IO file(s) (1 file(s) / 8192MB)

 

[Yves]

oh... so sorry.. shame on me that would have been easy... running it now. will edit this post in a few minutes...

 

[Yves]

do you need more tests?

 

[Rand__]

A similar writetest please
./fio --filename=test --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --ioengine=libaio --bs=4k --rwmixread=0 --iodepth=1 --numjobs=1 --runtime=60 --group_reporting --name=4kwritetest (adjust to win o/c)

and if you still are motivated, read and write with both 16 (depth/jobs) --iodepth=16 --numjobs=16

 

[Yves]

here you go

 

[Yves]

values seam to me for 1x 10GBit iSCSi connection okish...

 

[Rand__]

Have you seen that edited line?

"and if you still are motivated, read and write with both 16 (depth/jobs) --iodepth=16 --numjobs=16"

But yes it does look fine for the hardware involved.
O/C you could create a second volume to see what the optane's can do

 

[Yves]

Have you seen that edited line?

"and if you still are motivated, read and write with both 16 (depth/jobs) --iodepth=16 --numjobs=16"

But yes it does look fine for the hardware involved.
O/C you could create a second volume to see what the optane's can do

Nope, sorry missed it... doing it now....

write:

read:

I think it will not make any diffrence at all to the nvme since I think its the iscsi who is limiting the performance currently....

 

[Rand__]

Great, thanks a million.

I think it will not make any diffrence at all to the nvme since I think its the iscsi who is limiting the performance currently....

not on single thread iops
you have 3296 write iops single thread (and 95k on 16/16)
Nice values in the multi setup but o/c quite a lot of room for improvement on single.
Totally depends on your later use case o/c

 

[Yves]

Great, thanks a million.




not on single thread iops
you have 3296 write iops single thread (and 95k on 16/16)
Nice values in the multi setup but o/c quite a lot of room for improvement on single.
Totally depends on your later use case o/c

Sooo.... I got some more spare time. I will do now some more testing. Dual 10Gbit iSCSi... and I think I will also connect the iSCSi target to my C7000 compute node cluster to see how it will perform on that end... Maybe VMware is faster with handeling iSCSi than Windows

 

[Rand__]

Finally made some progress myself ... turns out that the order of nics passed on to the Starwind VSA is not the order it is visible in the GUI (reverse it seems) and also the config inside the VSA does not allow VLANs either (nor are they on *any*)
So now got a volume on a single box (have not found how to do HA yet). Have not managed to create a datastore on it though - timeouts

 

[Yves]

Finally made some progress myself ... turns out that the order of nics passed on to the Starwind VSA is not the order it is visible in the GUI (reverse it seems) and also the config inside the VSA does not allow VLANs either (nor are they on *any*)
So now got a volume on a single box (have not found how to do HA yet).

Oh... the Starwind guys told me... stay away from the VSA... its still beta... final version is planed for the 2nd half of this year. I think...

 

[Rand__]

Hm ok. So better to set up a pair of windows servers then. yay, after half a day to get it to go

 

[Yves]

Sorry, you should have started with I ran into some network troubles with the VSA... would have spared you sometime. But I also had to kill my both win2k16 machines... NICs started to act very very strange. Disableing stop working toke 5mins to reenable... or add a fixed ip... so I kill both reinstalled and I am back to try one more time... I also tought maybe if I am fast enough I will give PetaSAN a spin... looks also interessting.

 

[Rand__]

Yeah, gave me the opportunity to fix my malfunctioning second switch, so not totally wasted
Settting up starwind on windows now, will see how that goes

 

[Rand__]

Alright, so after finding out that windows doesn't like VLAN assignment on NIC level if its already assigned at vmware switch level I managed to get it to run. Single box with 2 nvme drives, 1 raw device mapping (Intel 750), left, one P3700 w/local datastore (right side)


Connected via 10GBe to remote server.No special settings (Raw device had write through), no extra read cache configured just Next Next Next. Jumbo frames on, rest default vmxnet3. No nvme drivers on the box, thats something i could try.

Major difference - raw device took about 75% CPU of the 3Ghz Xeon, datastore was <10%.
Most likely I could tweak reads a bit and maybe also cache writes on raw device (write back) but in the end still not as good as I had hoped.


Edit: New nvme driver gave slight improvement to P3700 result but nothing major.

 

[Yves]

Okay, so this is on top of a VMware ESXi 6.5U1? Did change from the vmware nvme driver to the intel nvme driver? What are the values if you benchmark those devices directly? Without Starwind vSAN?

 

[Rand__]

Actually I changed from microsoft nvme to intel nvme inside the starwind vm.
Native nvme/datastore was significant higher, will rerun. And yes 6.5 - need to check the actual release, maybe a couple of months old on that box

 

[Yves]

Did you passthru the nvme to the Windows VM or did you use a datastore on that NVME? I ask because on the VMware the default nvme driver is also not optimal for the Intel NVME...