I cannot confirm any problems with OmniOS 151028.
So I would -
- try another OmniOS (ex my ova template v4 if you use an older one, v4 comes with newest fixes)
Hi Gea,
had some time today to poke around my setup and installed v4 OVA (it was v3 I was using before BTW)... same HW specs as before (24 GB RAM):
with default tuning applied my localhost iperf results are impressive:
Code:
root@batcavefs:~# iperf -c 127.0.0.1 -w 1M
Connecting to host 127.0.0.1, port 5201
[ 4] local 127.0.0.1 port 57884 connected to 127.0.0.1 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 6.75 GBytes 58.0 Gbits/sec
[ 4] 1.00-2.00 sec 6.56 GBytes 56.3 Gbits/sec
[ 4] 2.00-3.00 sec 7.30 GBytes 62.7 Gbits/sec
[ 4] 3.00-4.00 sec 6.58 GBytes 56.5 Gbits/sec
[ 4] 4.00-5.00 sec 7.33 GBytes 62.9 Gbits/sec
[ 4] 5.00-6.00 sec 7.21 GBytes 61.9 Gbits/sec
[ 4] 6.00-7.00 sec 7.25 GBytes 62.3 Gbits/sec
[ 4] 7.00-8.00 sec 7.26 GBytes 62.4 Gbits/sec
[ 4] 8.00-9.00 sec 7.35 GBytes 63.1 Gbits/sec
[ 4] 9.00-10.00 sec 7.01 GBytes 60.2 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 70.6 GBytes 60.6 Gbits/sec sender
[ 4] 0.00-10.00 sec 70.6 GBytes 60.6 Gbits/sec receiver
Unfortunately iperf tests from other VM's again seems to suffer greatly in throughput and consistency vs old omniOS. Quite frustrating as I really do not understand why the old VM performs so much better.
iperf3 test from VMXnet3 Win10 VM to Omnios_v4 OVA:
Code:
C:\iperf-3.1.3-win64>iperf3.exe -c 10.10.1.2 -w 1M
Connecting to host 10.10.1.2, port 5201
[ 4] local 10.10.10.10 port 50003 connected to 10.10.1.2 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 1.20 GBytes 10.3 Gbits/sec
[ 4] 1.00-2.00 sec 1.25 GBytes 10.7 Gbits/sec
[ 4] 2.00-3.00 sec 1.25 GBytes 10.8 Gbits/sec
[ 4] 3.00-4.00 sec 1.25 GBytes 10.8 Gbits/sec
[ 4] 4.00-5.00 sec 1.37 GBytes 11.7 Gbits/sec
[ 4] 5.00-6.00 sec 1.27 GBytes 10.9 Gbits/sec
[ 4] 6.00-7.00 sec 1.25 GBytes 10.7 Gbits/sec
[ 4] 7.00-8.00 sec 1.17 GBytes 10.0 Gbits/sec
[ 4] 8.00-9.00 sec 1.27 GBytes 10.9 Gbits/sec
[ 4] 9.00-10.00 sec 1.28 GBytes 11.0 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 12.6 GBytes 10.8 Gbits/sec sender
[ 4] 0.00-10.00 sec 12.6 GBytes 10.8 Gbits/sec receiver
Still have lower throughput though with the latest OmniOS OVA:
Writing this file to the server from the VM was 50-90 MB/s, reading starts off half decent around 230-250 MB/s and then falls on it's face to 110-115 MB/s for the remainder of the transfer. Neither of this shows up on the old napp-it VM.
I have yet to try the e1000 and opem-vm-tools suggestion... I'll try to get to that next. But I don't see why any of this is happening. Now I'm curious to see if this would happen on my new server as well. Unfortunately it's not easy for me to connect my disks to it right now to import it. Making a new pool is not an option as I don't have other disks available at this time. I suppose I can check iperf3 tests from the windows VM
More testing:
iperf3 from my mediaserver Ubuntu 16:
Testing via the main vSwitch (normal network traffic):
Code:
root@mediasrv1:~# iperf3 -c 10.10.1.2 -w 1M
Connecting to host 10.10.1.2, port 5201
[ 4] local 10.10.1.10 port 36828 connected to 10.10.1.2 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 807 MBytes 6.77 Gbits/sec 0 492 KBytes
[ 4] 1.00-2.00 sec 968 MBytes 8.12 Gbits/sec 0 492 KBytes
[ 4] 2.00-3.00 sec 1010 MBytes 8.47 Gbits/sec 0 492 KBytes
[ 4] 3.00-4.00 sec 1011 MBytes 8.48 Gbits/sec 0 492 KBytes
[ 4] 4.00-5.00 sec 939 MBytes 7.87 Gbits/sec 0 492 KBytes
[ 4] 5.00-6.00 sec 975 MBytes 8.18 Gbits/sec 0 492 KBytes
[ 4] 6.00-7.00 sec 996 MBytes 8.36 Gbits/sec 0 492 KBytes
[ 4] 7.00-8.00 sec 985 MBytes 8.27 Gbits/sec 0 492 KBytes
[ 4] 8.00-9.00 sec 963 MBytes 8.08 Gbits/sec 0 492 KBytes
[ 4] 9.00-10.00 sec 983 MBytes 8.25 Gbits/sec 0 492 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 9.41 GBytes 8.09 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 9.41 GBytes 8.09 Gbits/sec receiver
Testing on storagenet vswitch:
Code:
root@mediasrv1:~# iperf3 -c 10.10.16.2 -w 1M
Connecting to host 10.10.16.2, port 5201
[ 4] local 10.10.16.10 port 55856 connected to 10.10.16.2 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 358 MBytes 3.00 Gbits/sec 0 491 KBytes
[ 4] 1.00-2.00 sec 650 MBytes 5.45 Gbits/sec 0 491 KBytes
[ 4] 2.00-3.00 sec 714 MBytes 5.99 Gbits/sec 0 491 KBytes
[ 4] 3.00-4.00 sec 664 MBytes 5.57 Gbits/sec 0 491 KBytes
[ 4] 4.00-5.00 sec 707 MBytes 5.93 Gbits/sec 0 491 KBytes
[ 4] 5.00-6.00 sec 717 MBytes 6.02 Gbits/sec 0 491 KBytes
[ 4] 6.00-7.00 sec 706 MBytes 5.92 Gbits/sec 0 491 KBytes
[ 4] 7.00-8.00 sec 726 MBytes 6.09 Gbits/sec 0 491 KBytes
[ 4] 8.00-9.00 sec 710 MBytes 5.95 Gbits/sec 0 491 KBytes
[ 4] 9.00-10.00 sec 714 MBytes 5.99 Gbits/sec 0 491 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 6.51 GBytes 5.59 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 6.51 GBytes 5.59 Gbits/sec receiver
Testing to new server (new hardware 2x 2695 V2's) connected via 10Gbe:
Code:
root@mediasrv1:~# iperf3 -c 10.10.10.34 -w 1M
Connecting to host 10.10.10.34, port 5201
[ 4] local 10.10.1.10 port 47954 connected to 10.10.10.34 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 555 MBytes 4.66 Gbits/sec 0 479 KBytes
[ 4] 1.00-2.00 sec 810 MBytes 6.80 Gbits/sec 0 479 KBytes
[ 4] 2.00-3.00 sec 787 MBytes 6.60 Gbits/sec 0 479 KBytes
[ 4] 3.00-4.00 sec 709 MBytes 5.95 Gbits/sec 0 479 KBytes
[ 4] 4.00-5.00 sec 746 MBytes 6.26 Gbits/sec 0 479 KBytes
[ 4] 5.00-6.00 sec 738 MBytes 6.19 Gbits/sec 0 479 KBytes
[ 4] 6.00-7.00 sec 769 MBytes 6.45 Gbits/sec 0 479 KBytes
[ 4] 7.00-8.00 sec 734 MBytes 6.15 Gbits/sec 0 479 KBytes
[ 4] 8.00-9.00 sec 639 MBytes 5.36 Gbits/sec 0 479 KBytes
[ 4] 9.00-10.00 sec 661 MBytes 5.54 Gbits/sec 0 479 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 6.98 GBytes 6.00 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 6.98 GBytes 6.00 Gbits/sec receiver
Now the same test w/ the old napp-it VM (OmniOS 151014) from mediaserver:
Code:
root@mediasrv1:~# iperf -c 10.10.1.2 -w 1M
------------------------------------------------------------
Client connecting to 10.10.1.2, TCP port 5001
TCP window size: 416 KByte (WARNING: requested 1.00 MByte)
------------------------------------------------------------
[ 3] local 10.10.1.10 port 42060 connected with 10.10.1.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 4.44 GBytes 3.81 Gbits/sec
root@mediasrv1:~# iperf -c 10.10.1.2 -w 1M -P 10
------------------------------------------------------------
Client connecting to 10.10.1.2, TCP port 5001
TCP window size: 416 KByte (WARNING: requested 1.00 MByte)
------------------------------------------------------------
[ 11] local 10.10.1.10 port 42120 connected with 10.10.1.2 port 5001
[ 3] local 10.10.1.10 port 42104 connected with 10.10.1.2 port 5001
[ 5] local 10.10.1.10 port 42106 connected with 10.10.1.2 port 5001
[ 4] local 10.10.1.10 port 42102 connected with 10.10.1.2 port 5001
[ 7] local 10.10.1.10 port 42110 connected with 10.10.1.2 port 5001
[ 6] local 10.10.1.10 port 42108 connected with 10.10.1.2 port 5001
[ 8] local 10.10.1.10 port 42112 connected with 10.10.1.2 port 5001
[ 9] local 10.10.1.10 port 42114 connected with 10.10.1.2 port 5001
[ 10] local 10.10.1.10 port 42116 connected with 10.10.1.2 port 5001
[ 12] local 10.10.1.10 port 42118 connected with 10.10.1.2 port 5001
^C[ ID] Interval Transfer Bandwidth
[ 6] 0.0- 8.1 sec 964 MBytes 993 Mbits/sec
[ 10] 0.0- 8.1 sec 819 MBytes 843 Mbits/sec
[ 11] 0.0- 8.1 sec 1006 MBytes 1.04 Gbits/sec
[ 3] 0.0- 8.1 sec 851 MBytes 876 Mbits/sec
[ 5] 0.0- 8.2 sec 946 MBytes 974 Mbits/sec
[ 4] 0.0- 8.1 sec 853 MBytes 878 Mbits/sec
[ 7] 0.0- 8.1 sec 834 MBytes 859 Mbits/sec
[ 8] 0.0- 8.2 sec 816 MBytes 839 Mbits/sec
[ 9] 0.0- 8.1 sec 835 MBytes 860 Mbits/sec
[ 12] 0.0- 8.1 sec 905 MBytes 932 Mbits/sec
[SUM] 0.0- 8.2 sec 8.62 GBytes 9.08 Gbits/sec
test from Win10 VM:
Code:
C:\iperf-2.0.5-2-win32>iperf -c 10.10.1.2
------------------------------------------------------------
Client connecting to 10.10.1.2, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.10.10.10 port 50044 connected with 10.10.1.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 240 MBytes 201 Mbits/sec
C:\iperf-2.0.5-2-win32>iperf -c 10.10.1.2 -w 5M
------------------------------------------------------------
Client connecting to 10.10.1.2, TCP port 5001
TCP window size: 5.00 MByte
------------------------------------------------------------
[ 3] local 10.10.10.10 port 50045 connected with 10.10.1.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 4.36 GBytes 3.74 Gbits/sec
C:\iperf-2.0.5-2-win32>iperf -c 10.10.1.2 -w 1M
------------------------------------------------------------
Client connecting to 10.10.1.2, TCP port 5001
TCP window size: 1.00 MByte
------------------------------------------------------------
[ 3] local 10.10.10.10 port 50046 connected with 10.10.1.2 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 4.50 GBytes 3.86 Gbits/sec