ZFS vs. ESXI VM Datastore

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,004
1,569
113
CA
HGST 200GB 12Gb/s SAS Drive connected to LSI 3008 HBA on my SM motherboard.
(Note; I didn't flash to IT firmware or update firmware so it's whatever came on it at the moment.)


Drive used as ESXI data store with windows 10 VM on it, and benchmarked via CDM.
esxi.png

LSI3008 passed-through to Napp-IT (OmniOS) VM running ZFS. Single drive, sync=disabled, record=4k

Left = Default VM Setup (E1000, no jumbo, no tuning tcp packet)
RIght = VMXNET3, MTU=9000 on vswitch & omnios & tcp packet increase

compare.jpg

The reads are now practically identical to the ESXI Datastore (not passed) but the writes are still 50% of the direct datastore in esxi. (Note: Sync=off).

Anyone have tips on the writes?
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,004
1,569
113
CA
Here's Anvil IOPs testing of before and after network tweaking.
anvil-sas1.jpg anvil-sas2.jpg

Some really good increases at 4K but at higher QD barely any... is this likely due to increased packet size?

For reference when passed through to Win10 VM (no os on this drive) the IOPs are MUCH higher. (I don't have a screenshot/test of running Win10 on this drive with Anvil but it is likely onyl 10-15% perf. drop at high QD not 50% we're seeing with ZFS)
sas3.jpg
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,004
1,569
113
CA
Also note the latency between network configurations and direct pass through to VM rather insane although not to be unexpected with the data flow would be nice to drop a bit more too :D
 

gea

Well-Known Member
Dec 31, 2010
2,485
837
113
DE
Jumbo frames should not be relevant as transfers between ESXi and NFS are internal in software.
Tcp/ip parameters are relevant for a real network.

Important are the vmxnet3s settings, ex (may require an ESXi reboot)
TxRingSize=1024,1024,1024,1024,1024,1024,1024,1024,1024,1024;
RxRingSize=1024,1024,1024,1024,1024,1024,1024,1024,1024,1024;
RxBufPoolLimit=4096,4096,4096,4096,4096,4096,4096,4096,4096,4096;
EnableLSO=0,0,0,0,0,0,0,0,0,0;

and NFS service tunings like
set nfs:nfs3_max_threads=1024
set nfs:nfs4_max_threads=1024
* increase NFS read ahead count
set nfs:nfs3_nra=32
set nfs:nfs4_nra=32
* increase NFS maximum transfer size
set nfs3:max_transfer_size=1048576
set nfs4:max_transfer_size=1048576
* increase NFS logical block size
set nfs:nfs3_bsize=1048576
set nfs:nfs4_bsize=1048576

These settings are part of my default tuning set in newest napp-it
You can try these settings or modify there with other values.

Overall, ZFS is not as fast as ext/ntfs/vmfs due the extra checksumming and CopyOnWrite
but much safer - but with the advanced caching features and as it scales very good over disks and vdevs ZFS can give a better performance despite that.
 
Last edited:
  • Like
Reactions: TechTrend

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,004
1,569
113
CA
@gea that is what I thought too (and honestly why I never changed any settings) but doing tests in Win10 and with Iperf between OmniOS and CentOS I got 3.8Gbit after changing TCP settings I got that to 15.x Gbit to OmniOS from Linux.

I was getting 18Gbit to CentOS from OmniOS but when I changed TCP it dropped to 15Gbit (still acceptable).

CentOS and OmniOS are both VMs on my AIO ESXI host, so changing TCP does appear to make a rather big difference even on the vSwitch.

I'm going to update Napp-IT now and try to tweak some more, thank you for all the advice and tips! Hopefully I can help boost these #s even more.

(For record my OmniOS/Napp-IT VM is allocated all cores, and does shoot up high CPU usage too, and has 32GB reserved RAM, might need to bump that to 64 when doing these tests since it's moving data so fast between NVME and SAS12 the timed tests are using ~20GB)
 

_alex

Active Member
Jan 28, 2016
874
94
28
Bavaria / Germany
I think the tcp/ip stack in the guests and also cpu-cycles from zfs become visible here.
What Numbers would you get when you export a ramdisk from one vm to the other? once with zfs, another time as nfs? could help to get an idea what is possible in the best case / tune the tcp/ip settings.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,004
1,569
113
CA
Here's CPU utilization:
I should have also noted this was NFS->ESXI and bench ran in win10 VM on the NFS share
ScreenClip.png

Here is after Napp-IT Pro update to latest version, no tuning yet.
ScreenClip.png

ScreenClip.png
Some things dropped, some went up, I'm thinking nothing really changed between versions re: performance.
There's a couple things in tuning slightly different than what I have now, we'll see if that matters :)

Still only 50% IOPs as seen with pass-through to Windows VM direct... hoping to get IOPs up, but not sure if that will be possible with ZFS in general. 50% sure seems like a huge hit.
 

gea

Well-Known Member
Dec 31, 2010
2,485
837
113
DE
napp-it release is not performance relevant.
This is different to the underlying OS release ex 151014 or 151018 (propably tomorrow).

But tuning is essential as the defaults are optimized to 1G netwoks and low RAM,
especially the vmxnet3 and nfs settings in an AiO setup.

With a real 10G ethernet or SMB, you should increase tcp buffers and go to jumboframes,
http://napp-it.de/doc/downloads/performance_smb2.pdf
 
  • Like
Reactions: T_Minus

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,004
1,569
113
CA
Here's my current settings vs. the new 'tuning' options.

ScreenClip.png

Looks like the NFS Max Threads goes up a lot from 8 to 1024. The TCP/Jumboframe stuff is appearing 2x what mine is set at currently for recv and send and some vmxnet3 tweaks, hopefully we see some more IOps :)