Solaris network slow (vmxnet3)

Rand__

Well-Known Member
Mar 6, 2014
4,497
878
113
I am trying to see why Napp-IT does not seem to deliver what the array could be capable of. So I started on Omni OS and was not happy and now have switched to Solaris, but have similar issues there...

Since I could find no reasons why it didnt work I moved from testing (VM2VM) network to local host and even that is not working. I have a VM with 2 CPUs (2.4 Ghz), 4GB Ram and Solaris 11.4. I have applied Napp-It tuning, have vmxnet3 and have installed ESX6 vmware tools (newest official ones I could find).

I run iperf 2.04 to localhost using window size 1M (which is faster than 2M default):
Code:
root@SOL114:/tmp# /opt/csw/bin/iperf -c 127.0.0.1 -w 1M
------------------------------------------------------------
Client connecting to 127.0.0.1, TCP port 5001
TCP window size: 1.00 MByte
------------------------------------------------------------
[  3] local 127.0.0.1 port 61739 connected with 127.0.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  7.65 GBytes  6.56 Gbits/sec
As a comparison value I run the same on the ESXi host:
Code:
/opt/iperf/bin/iperf -c 127.0.0.1 -w 1M
------------------------------------------------------------
Client connecting to 127.0.0.1, TCP port 5001
TCP window size: 1.01 MByte (WARNING: requested 1.00 MByte)
------------------------------------------------------------
[  3] local 127.0.0.1 port 15605 connected with 127.0.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  35.4 GBytes  30.4 Gbits/sec
And for the fun of it on my FreeNas Host (11 U6, 4 CPUs, 32 GB Ram)
Code:
root@freenas6:~ # iperf -c 127.0.0.1
------------------------------------------------------------
Client connecting to 127.0.0.1, TCP port 5001
TCP window size: 47.9 KByte (default)
------------------------------------------------------------
[  3] local 127.0.0.1 port 61403 connected with 127.0.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  41.6 GBytes  35.7 Gbits/sec

So I really wonder - why does Solaris have so bad performance? Or rather - any idea how to fix it?:)
 

gea

Well-Known Member
Dec 31, 2010
2,489
838
113
DE
There must be another problem.
I made a test with OmniOS 151028 and the included iperf with the result (ESXi 6.7u1)

iperf 127.0.0.1: 48,6 Gb/s
iperf ESXi internal between two OmniOS VMs with OS defaults: 7,7 Gb/s
iperf ESXi internal between two OmniOS VMs with a napp-it default tuning: 9 Gb/s
 

Attachments

Rand__

Well-Known Member
Mar 6, 2014
4,497
878
113
Hm, fresh Solaris install... no optimization, no vmware tools.

Code:
root@sol1142:~# /opt/csw/bin/iperf -c 127.0.0.1 -w 1M
------------------------------------------------------------
Client connecting to 127.0.0.1, TCP port 5001
TCP window size: 1.00 MByte
------------------------------------------------------------
[  3] local 127.0.0.1 port 55117 connected with 127.0.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  13.2 GBytes  11.3 Gbits/sec
root@sol1142:~# /opt/csw/bin/iperf -c 127.0.0.1 -w 1M -P 4
------------------------------------------------------------
Client connecting to 127.0.0.1, TCP port 5001
TCP window size: 1.00 MByte
------------------------------------------------------------
[  4] local 127.0.0.1 port 43197 connected with 127.0.0.1 port 5001
[  3] local 127.0.0.1 port 60271 connected with 127.0.0.1 port 5001
[  5] local 127.0.0.1 port 58374 connected with 127.0.0.1 port 5001
[  6] local 127.0.0.1 port 37858 connected with 127.0.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  5.43 GBytes  4.67 Gbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  5.55 GBytes  4.77 Gbits/sec
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec  5.55 GBytes  4.77 Gbits/sec
[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-10.0 sec  3.64 GBytes  3.13 Gbits/sec
[SUM]  0.0-10.0 sec  20.2 GBytes  17.3 Gbits/sec
Better but not good.

Are there any other vmware tools that I can install than the ESX 6.0 official release? I looked at the open vm tools but couldnt find those for solaris...
 

Rand__

Well-Known Member
Mar 6, 2014
4,497
878
113
So I moved the vm over to another box and performance is better there... not really good though

Code:
root@sol1142:~# /opt/csw/bin/iperf -c 127.0.0.1
------------------------------------------------------------
Client connecting to 127.0.0.1, TCP port 5001
TCP window size: 48.0 KByte (default)
------------------------------------------------------------
[  3] local 127.0.0.1 port 44456 connected with 127.0.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  23.6 GBytes  20.3 Gbits/sec
root@sol1142:~# /opt/csw/bin/iperf -c 127.0.0.1 -w 1M -P 4
------------------------------------------------------------
Client connecting to 127.0.0.1, TCP port 5001
TCP window size: 1.00 MByte
------------------------------------------------------------
[  6] local 127.0.0.1 port 41835 connected with 127.0.0.1 port 5001
[  5] local 127.0.0.1 port 43323 connected with 127.0.0.1 port 5001
[  3] local 127.0.0.1 port 52667 connected with 127.0.0.1 port 5001
[  4] local 127.0.0.1 port 40076 connected with 127.0.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-10.0 sec  8.08 GBytes  6.94 Gbits/sec
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  5.61 GBytes  4.82 Gbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  6.13 GBytes  5.27 Gbits/sec
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec  7.46 GBytes  6.41 Gbits/sec
[SUM]  0.0-10.0 sec  27.3 GBytes  23.4 Gbits/sec
I then also moved my original VM:
Code:
root@SOL114:~# /opt/csw/bin/iperf -c 127.0.0.1
------------------------------------------------------------
Client connecting to 127.0.0.1, TCP port 5001
TCP window size: 2.00 MByte (default)
------------------------------------------------------------
[  3] local 127.0.0.1 port 60182 connected with 127.0.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  22.5 GBytes  19.3 Gbits/sec
So on the second host the vm internal iperf is 5 times faster then on the original host, but still 3 times slower than a reference value (Geas solaris vm, and also my omni os vm on that host which gets 66GB/s locally).

Still no clue why and how to improve.

Vm to external VM is still limited at around 10Gbit/s too... makes me wonder whether I need to move back to physical boxes after all to get >10G speeds
 

gea

Well-Known Member
Dec 31, 2010
2,489
838
113
DE
Performance of a virtualized nic and switch depend on hardware.
I suppose if you want to go beyond 10G you need a real nic in pass-through mode or you indeed need a physical server.