Does this 10GBe performance look right?

[Kev]

I have a A1SRi-2758F , 32GB DDR, Samsung 845DC EVO MZ-7GE960 as local datastore right now, ConnectX-4 LX running ESXI 6.5 U1 and I created a freenas VM serving up two SanDisk SDSSDH31000G mirrored as my ESXI datastore for other ESXI machines. Performance is only hitting around ~150MB/sec so I started looking into things.

I came to the conclusion that the network interface is the bottleneck. I using a vmxnet3 adapter in Freenas and have already done a lot go tweaking like 4 RX/TX queues, network stack buffer optimizations and all I am gettin from the ESXI shell to Freenas running iPerf is around ~1.3Gbit/sec and maybe around ~1.3 to 3GBit/sec when testing with iPerf and -P 4 (4 threads).

I fired up a windows 10 machine and I am getting similar low speeds from ESXI to VM. When I do ESXI to ESXI, I get easily 8-9GBit/sec but I can't understand why the vmxnet3 interface is underperforming. When I look at top -SH on my freenas machine, I can see intr process taking a lot of CPU time.

Is this what I should expect from the C2758 architecture? Thanks for looking!

 

[Net-Runner]

Since you have pretty valid Mellanox stuff for networking and Atom CPU (which is not quite powerful) I would strongly recommend you using something that supports RDMA (iSER for iSCSI). This technology offloads IOps from CPU to network/memory and usually consumes fewer resources for storage traffic compared to traditional protocols. Seems like free Starwind Software Defined Storage for the HCI • StarWind Virtual SAN ® Free supports it StarWind iSER so you might be interested to give it a try.

 

[_alex]

why not passthrough a vf via sr-iov from the cx4 into FreeNAS?
doesn't work on ESXi with cx3 but should be possible with cx4.
my feeling is that the Atom might be a bit on the low side for such a setup, but should be able to saturate 10gbe with iperf. have you used -P >1 with iperf?

 

[Kev]

So Atom C2000 does not support VT-d. I did try iperf -P 4 and set up 4 RX/TX queues but that only got around 2.8Gbps. I'm not using OMV with ZFS plugin because when I tried vanilla Debian, I could get around 4Gbps using only 1 iperf stream. I just don't think the VMXNET3 drivers are very well tuned for FreeBSD.

I'm looking into Starwind virtual san free but I must have a windows license to install this? Is there no virtual appliance I can deploy?

 

[K D]

StarWind Virtual Storage Appliance