ESXi AiO VM backed on NFS slow r/w

[ARNiTECT]

Hi,

I am in the process of setting up 10Gbe on my network, the process hasn't been straight forward, but I'm poring over forums trying to get the best out of my All-in-One server. The trouble I'm having is with VMs on the same server only reading & writing at 1Gb/s, when physical connections to devices outside the server are closer to 10Gb/s.

Host is ESXi 7.0 U2, Storage VM is OmniOS r151040l with napp-it 21.06, sharing: NFS, SMB, iSCSI, 8x vCPU and 48GB ram.

I'm struggling with slow Windows 10 VMs backed on the NFS datastores, on a fast pool of 3x NVMe Z1, sync disabled. CrystalDiskMark reports c: drive r/w: 100MB/s only, which is much lower than expected.

The same VM backed on a single local SSD performs as expected CrystalDiskMark c: drive r/w:450MB/s (MX500-1TB)

During CDM benchmark on the NFS backed VM, OmniOS CPU load is low, Pool use is low, wait is low
iPerf3 from Win VM to OmniOS VM is 23Gb/s, which I would expect for vmxnet3 over software

iSCSI tests over physical network are very good with CrystalDiskMark MTU 9000: w:650MB/s & r:1050MB/s from targets on a range of pools (Optane 900p, MP510, HDD)
SMB 'reads' appear much slower than iSCSI (MTU 1500: w:700MB/s & r:300MB/s and MTU 9000: w:900MB/s & r:450MB/s), but this is another matter.

The OmniOS/napp-it 'base' tuning is applied and other NFS and SMB properties are set to defaults.

I have run out of ideas of what to adjust in ESXi (latest VMtools, ethernet0.coalescingScheme = disabled), but as I am getting good speed over a physical network, I assumed it must be something in ESXi.

Any thoughts welcome.

 

[TRACKER]

Hi Arni,
i saw also your thread on hardforum
I've been testing iscsi and nfs as underlying storage for VM on solaris 11.x and esxi as hypervisor.
During my testing results with nfs were so bad, compared to iscsi, that i decided to completely ditch nfs as possible vm storage on solaris.
Fortunately for me, recently i completely migrated my data (NAS/SAN) from solaris 11.4 to truenas and i think that was one of the best decisions i ever took regarding how and on what software platform i store my data.
It seems on omnios you have something similar.

I know Gea will have some comments against my reply here, but i will say it anyway
Just forget about using nfs as underlying storage for vm.
At least on omnios...

 

[ARNiTECT]

Hi Tracker,
Yes, I am reaching out to Gea for tips on what could be causing the speed cap from the side of OmniOS/napp-it.
I have done some tests on my secondary server, which is currently running filesystems replicated from my main server. This is achieving good speeds of r:500MB/s & w:400MB/s, on NFS backed VMs (4x 250GB MX500 SSD Raid10 with sync enabled).
I should spin up a VM of TrueNAS Core, as I've not tried it. I first used napp-it for an NFS all-in-one on Open Indiana a long time ago and then moved on to OmniOS; I don't remember having issues with NFS until my recent tests for setting up 10Gbe. ...I'm considering a complete reinstall. I hope this is a minor blip on my path to home server nirvana.

 

[ARNiTECT]

I did a complete reinstall of the napp-it VM, but unfortunately, the VM backed on NFS drive is still capped at around 100MB/s.
I haven't reinstalled ESXi yet; however, VM backed on local SSD is still around 450MB/s
This 100-110MB/s seems like maxed out 1Gbe speeds; I just wonder if somewhere in the ESXi settings I've foolishly setup something incorrectly.

 

[Rand__]

u set vmxnet3 as nic type?

 

[ARNiTECT]

u set vmxnet3 as nic type?

There are 2x vmxnet3 nics on the OmniOS/napp-it VM
- vnicA attached to vSwitchA with all other VMs, 10Gbe physical port A, all MTU 1500
- vnicB on different subnet, attached to vSwitchB, 10Gbe physical port B direct cable to windows 10 pc portB, all MTU 9000 (for iSCSI)
/kernel/drv/ vmxnet3s.conf MTU=1500,9000,1500,1500,1500,1500,1500,1500,1500,1500;

 

[gea]

iSCSI vs NFS
I would expect a quite similar performance from both if you care about the different default sync behaviour. At least with ESXi the default sync write behaviour of NFS is sync enabled while on iSCSI the corresponding default write back behaviour is like sync=off.

If you want to compare both, set sync=on on NFS and writeback=off on iSCSI or the other setting on either. I would also not expect a huge difference between Solaris/OmniOS and Free-BSD. On both defaults are more optimized for 1G and for 10G you need some tunings. The minimalistic Solaris text or OmniOS are among the fastest server options for iSCSI, NFS or SMB.

 

[ARNiTECT]

Woo hoo!
NFS backed VM CrystalDiskMark c: drive r:1000MB/s w:700MB/s

It was something in my ESXi Networking settings that I had messed up in my pursuit of 10Gbe, but not exactly sure what.
To fix this: I deleted an unused port group, and it was immediately replaced with 'VM Network'; this was a eureka moment where I must have messed up something. I removed all VMs from a port group I had created for my new 10Gbe X710-DA2 nic, with an intention to separate VMs from management; deleted the new vSwitch1 used with that group; added the now unused X710 port to the vSwitch0; both 'VM Network' and 'Management Network' now using vSwitch0. I kept a port group I had created called 'Direct 10G' with 'vSwitch2' using the other X710 port for my direct cable from host to workstation for iSCSI with jumbo frames.

So after I tidy up the mess I've made with reinstalled VMs, I need to better understand how to setup ESXi networking.