Crap iperf3 performance in OmniOS?

[AveryFreeman]

Hey,

so I just set up vmxnet3 drivers in OmniOS new install, 151028b on ESXi 6.7U1.

Went to test throughput to see what the difference between the E1000 and vmxnet3 drivers were like, so I fired up iperf3 on pfsense as a server, and sent the most basic test from the OmniOS VM as a client using both of them.

I was shocked by what I found!

Here's vmxnet3:

# iperf3 -c 192.168.1.1
Connecting to host 192.168.1.1, port 5201
[  4] local 192.168.1.64 port 58558 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   104 MBytes   874 Mbits/sec
[  4]   1.00-2.00   sec  96.7 MBytes   811 Mbits/sec
[  4]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[  4]   3.00-4.00   sec  50.8 MBytes   426 Mbits/sec
[  4]   4.00-5.00   sec  50.1 MBytes   421 Mbits/sec
[  4]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[  4]   6.00-7.00   sec  52.9 MBytes   444 Mbits/sec
[  4]   7.00-8.00   sec  52.7 MBytes   442 Mbits/sec
[  4]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[  4]   9.00-10.00  sec  50.7 MBytes   425 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   458 MBytes   384 Mbits/sec                  sender
[  4]   0.00-10.00  sec   458 MBytes   384 Mbits/sec                  receiver

Here's the e1000:

# iperf3 -c 192.168.1.1
Connecting to host 192.168.1.1, port 5201
[  4] local 192.168.1.64 port 36558 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   104 MBytes   875 Mbits/sec
[  4]   1.00-2.00   sec  91.7 MBytes   770 Mbits/sec
[  4]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[  4]   3.00-4.00   sec  55.5 MBytes   465 Mbits/sec
[  4]   4.00-5.00   sec   112 MBytes   941 Mbits/sec
[  4]   5.00-6.00   sec  13.0 MBytes   109 Mbits/sec
[  4]   6.00-7.00   sec   144 KBytes  1.18 Mbits/sec
[  4]   7.00-8.00   sec  6.10 MBytes  51.2 Mbits/sec
[  4]   8.00-9.00   sec   192 KBytes  1.58 Mbits/sec
[  4]   9.00-10.00  sec  72.9 MBytes   612 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   456 MBytes   383 Mbits/sec                  sender
[  4]   0.00-10.00  sec   456 MBytes   382 Mbits/sec                  receiver

WTF?

Upon further inspection, I noticed some differences between the original vmxnets3.conf included with OmniOS (which I kept just in case) and the one installed with VMWare-Tools 10.25.0:

VMWare-Tools version:

root@omniosce:/kernel/drv# cat vmxnet3s.conf.original-vmware
# Driver.conf(4) file for VMware Vmxnet Generation 3 adapters.
# TxRingSize --
#
#    Tx ring size for each vmxnet3s# adapter. Must be a multiple of 32.
#
#    Minimum value: 32
#    Maximum value: 4096
#
TxRingSize=256,256,256,256,256,256,256,256,256,256;
# RxRingSize --
#
#    Rx ring size for each vmxnet3s# adapter. Must be a multiple of 32.
#
#    Minimum value: 32
#    Maximum value: 4096
#
RxRingSize=256,256,256,256,256,256,256,256,256,256;
# RxBufPoolLimit --
#
#    Limit the number of Rx buffers cached for each vmxnet3s# adapter.
#    Increasing the limit might improve performance but increases the
#    memory footprint.
#
#    Minimum value: 0
#    Maximum value: RxRingSize * 10
#
RxBufPoolLimit=512,512,512,512,512,512,512,512,512,512;
# EnableLSO --
#
#    Enable or disable LSO for each vmxnet3s# adapter.
#
#    Minimum value: 0
#    Maximum value: 1
#
EnableLSO=1,1,1,1,1,1,1,1,1,1;
# MTU --
#
#    Set MTU for each vmxnet3s# adapter.
#
#    Minimum value: 60
#    Maximum value: 9000
#
#MTU=1500,1500,1500,1500,1500,1500,1500,1500,1500,1500;

root@omniosce:/kernel/drv# cat vmxnet3s.conf
# Driver.conf(4) file for VMware Vmxnet Generation 3 adapters.
# TxRingSize --
#
#    Tx ring size for each vmxnet3s# adapter. Must be a multiple of 32.
#
#    Minimum value: 32
#    Maximum value: 4096
#
TxRingSize=4096,4096,4096,4096,4096,4096,4096,4096,4096,4096;
# RxRingSize --
#
#    Rx ring size for each vmxnet3s# adapter. Must be a multiple of 32.
#
#    Minimum value: 32
#    Maximum value: 4096
#
RxRingSize=4096,4096,4096,4096,4096,4096,4096,4096,4096,4096;
# RxBufPoolLimit --
#
#    Limit the number of Rx buffers cached for each vmxnet3s# adapter.
#    Increasing the limit might improve performance but increases the
#    memory footprint.
#
#    Minimum value: 0
#    Maximum value: RxRingSize * 10
#
RxBufPoolLimit=16384,16384,16384,16384,16384,16384,16384,16384,16384,16384;
# EnableLSO --
#
#    Enable or disable LSO for each vmxnet3s# adapter.
#
#    Minimum value: 0
#    Maximum value: 1
#
EnableLSO=1,1,1,1,1,1,1,1,1,1;

Interesting... going to reboot and load these included vmxnet3s.conf and vmxnet drivers and see if there's any difference...

 

[AveryFreeman]

OK - after reboot the results are about what I'd expect, around 922 Mbps for both E1000 and vmxnet3 - thank god! Those speeds were awful.

Although there's no real difference between the iperf results, I do suspect that the CPU might be less encumbered by the vmxnet3 driver since it's paravirtualized, so I'll probably stick with that.

 

[gea]

Did you use genuine vmware tools or the in Illumos distributions included open-vm-tools (pkg install open-vm-tools)?
With the last I got (they use increased vmxnet ring size and LSO=0 per default):

iperf 127.0.0.1: 48,6 Gb/s
iperf ESXi internal between two OmniOS VMs with OS defaults: 7,7 Gb/s
iperf ESXi internal between two OmniOS VMs with a napp-it default tuning: 9 Gb/s

Napp-it tuning, see menu System > Appliance tuning
If you modify system settings, you usually need a reboot

https://forums.servethehome.com/index.php?threads/solaris-network-slow-vmxnet3.22613/