Solaris network slow (vmxnet3)

Discussion in 'Solaris, Nexenta, OpenIndiana, and napp-it' started by Rand__, Nov 21, 2018.

  1. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,587
    Likes Received:
    542
    I am trying to see why Napp-IT does not seem to deliver what the array could be capable of. So I started on Omni OS and was not happy and now have switched to Solaris, but have similar issues there...

    Since I could find no reasons why it didnt work I moved from testing (VM2VM) network to local host and even that is not working. I have a VM with 2 CPUs (2.4 Ghz), 4GB Ram and Solaris 11.4. I have applied Napp-It tuning, have vmxnet3 and have installed ESX6 vmware tools (newest official ones I could find).

    I run iperf 2.04 to localhost using window size 1M (which is faster than 2M default):
    Code:
    root@SOL114:/tmp# /opt/csw/bin/iperf -c 127.0.0.1 -w 1M
    ------------------------------------------------------------
    Client connecting to 127.0.0.1, TCP port 5001
    TCP window size: 1.00 MByte
    ------------------------------------------------------------
    [  3] local 127.0.0.1 port 61739 connected with 127.0.0.1 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-10.0 sec  7.65 GBytes  6.56 Gbits/sec
    As a comparison value I run the same on the ESXi host:
    Code:
    /opt/iperf/bin/iperf -c 127.0.0.1 -w 1M
    ------------------------------------------------------------
    Client connecting to 127.0.0.1, TCP port 5001
    TCP window size: 1.01 MByte (WARNING: requested 1.00 MByte)
    ------------------------------------------------------------
    [  3] local 127.0.0.1 port 15605 connected with 127.0.0.1 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-10.0 sec  35.4 GBytes  30.4 Gbits/sec
    And for the fun of it on my FreeNas Host (11 U6, 4 CPUs, 32 GB Ram)
    Code:
    root@freenas6:~ # iperf -c 127.0.0.1
    ------------------------------------------------------------
    Client connecting to 127.0.0.1, TCP port 5001
    TCP window size: 47.9 KByte (default)
    ------------------------------------------------------------
    [  3] local 127.0.0.1 port 61403 connected with 127.0.0.1 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-10.0 sec  41.6 GBytes  35.7 Gbits/sec
    

    So I really wonder - why does Solaris have so bad performance? Or rather - any idea how to fix it?:)
     
    #1
  2. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,261
    Likes Received:
    750
    There must be another problem.
    I made a test with OmniOS 151028 and the included iperf with the result (ESXi 6.7u1)

    iperf 127.0.0.1: 48,6 Gb/s
    iperf ESXi internal between two OmniOS VMs with OS defaults: 7,7 Gb/s
    iperf ESXi internal between two OmniOS VMs with a napp-it default tuning: 9 Gb/s
     

    Attached Files:

    #2
  3. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,587
    Likes Received:
    542
    Hm, fresh Solaris install... no optimization, no vmware tools.

    Code:
    root@sol1142:~# /opt/csw/bin/iperf -c 127.0.0.1 -w 1M
    ------------------------------------------------------------
    Client connecting to 127.0.0.1, TCP port 5001
    TCP window size: 1.00 MByte
    ------------------------------------------------------------
    [  3] local 127.0.0.1 port 55117 connected with 127.0.0.1 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-10.0 sec  13.2 GBytes  11.3 Gbits/sec
    root@sol1142:~# /opt/csw/bin/iperf -c 127.0.0.1 -w 1M -P 4
    ------------------------------------------------------------
    Client connecting to 127.0.0.1, TCP port 5001
    TCP window size: 1.00 MByte
    ------------------------------------------------------------
    [  4] local 127.0.0.1 port 43197 connected with 127.0.0.1 port 5001
    [  3] local 127.0.0.1 port 60271 connected with 127.0.0.1 port 5001
    [  5] local 127.0.0.1 port 58374 connected with 127.0.0.1 port 5001
    [  6] local 127.0.0.1 port 37858 connected with 127.0.0.1 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  4]  0.0-10.0 sec  5.43 GBytes  4.67 Gbits/sec
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-10.0 sec  5.55 GBytes  4.77 Gbits/sec
    [ ID] Interval       Transfer     Bandwidth
    [  5]  0.0-10.0 sec  5.55 GBytes  4.77 Gbits/sec
    [ ID] Interval       Transfer     Bandwidth
    [  6]  0.0-10.0 sec  3.64 GBytes  3.13 Gbits/sec
    [SUM]  0.0-10.0 sec  20.2 GBytes  17.3 Gbits/sec
    
    Better but not good.

    Are there any other vmware tools that I can install than the ESX 6.0 official release? I looked at the open vm tools but couldnt find those for solaris...
     
    #3
  4. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,261
    Likes Received:
    750
    Curious, I get nearly 52 Gb/s

    iperf-solaris114.png
     
    #4
  5. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,587
    Likes Received:
    542
    So I moved the vm over to another box and performance is better there... not really good though

    Code:
    root@sol1142:~# /opt/csw/bin/iperf -c 127.0.0.1
    ------------------------------------------------------------
    Client connecting to 127.0.0.1, TCP port 5001
    TCP window size: 48.0 KByte (default)
    ------------------------------------------------------------
    [  3] local 127.0.0.1 port 44456 connected with 127.0.0.1 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-10.0 sec  23.6 GBytes  20.3 Gbits/sec
    root@sol1142:~# /opt/csw/bin/iperf -c 127.0.0.1 -w 1M -P 4
    ------------------------------------------------------------
    Client connecting to 127.0.0.1, TCP port 5001
    TCP window size: 1.00 MByte
    ------------------------------------------------------------
    [  6] local 127.0.0.1 port 41835 connected with 127.0.0.1 port 5001
    [  5] local 127.0.0.1 port 43323 connected with 127.0.0.1 port 5001
    [  3] local 127.0.0.1 port 52667 connected with 127.0.0.1 port 5001
    [  4] local 127.0.0.1 port 40076 connected with 127.0.0.1 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  6]  0.0-10.0 sec  8.08 GBytes  6.94 Gbits/sec
    [ ID] Interval       Transfer     Bandwidth
    [  4]  0.0-10.0 sec  5.61 GBytes  4.82 Gbits/sec
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-10.0 sec  6.13 GBytes  5.27 Gbits/sec
    [ ID] Interval       Transfer     Bandwidth
    [  5]  0.0-10.0 sec  7.46 GBytes  6.41 Gbits/sec
    [SUM]  0.0-10.0 sec  27.3 GBytes  23.4 Gbits/sec
    
    I then also moved my original VM:
    Code:
    root@SOL114:~# /opt/csw/bin/iperf -c 127.0.0.1
    ------------------------------------------------------------
    Client connecting to 127.0.0.1, TCP port 5001
    TCP window size: 2.00 MByte (default)
    ------------------------------------------------------------
    [  3] local 127.0.0.1 port 60182 connected with 127.0.0.1 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-10.0 sec  22.5 GBytes  19.3 Gbits/sec
    
    So on the second host the vm internal iperf is 5 times faster then on the original host, but still 3 times slower than a reference value (Geas solaris vm, and also my omni os vm on that host which gets 66GB/s locally).

    Still no clue why and how to improve.

    Vm to external VM is still limited at around 10Gbit/s too... makes me wonder whether I need to move back to physical boxes after all to get >10G speeds
     
    #5
  6. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,261
    Likes Received:
    750
    Performance of a virtualized nic and switch depend on hardware.
    I suppose if you want to go beyond 10G you need a real nic in pass-through mode or you indeed need a physical server.
     
    #6
  7. tomaash

    tomaash Member

    Joined:
    Oct 11, 2016
    Messages:
    87
    Likes Received:
    33
    You could also try SR-IOV.
     
    #7
  8. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,587
    Likes Received:
    542
    Yeah but Solaris only has IB support for MLX ;)
     
    #8
Similar Threads: Solaris network
Forum Title Date
Solaris, Nexenta, OpenIndiana, and napp-it Solaris 11.3 network slow... Feb 12, 2017
Solaris, Nexenta, OpenIndiana, and napp-it solaris 11.4 and samsung 960 evo Oct 29, 2019
Solaris, Nexenta, OpenIndiana, and napp-it OmniOS vs Solaris 11.4 SMB default behaviour Jun 6, 2019
Solaris, Nexenta, OpenIndiana, and napp-it Solaris 11.4 meta devices for DDT Apr 10, 2019
Solaris, Nexenta, OpenIndiana, and napp-it Should I use open-vm-tools instead of vmware-tools solaris? Mar 12, 2019

Share This Page