ok so toying with a new build and trying to get max perf out of it. If that works out fine then I might switch my vsan to dual ZFS boxes...
I will perform the initial tests on a X11DPI-T /6150's, 192GB Ram running Esxi 6.7U1. Bios is on the lastest SM version, Performance mode active in bios too.
I currently have a bunch of NVMe drives attached to it (Optanes, P3700's, Intel 750s) to have a proper basis for testing.
I got the OmniOS build and updated it to the latest version (currently omnios-r151028-d3d0427bff) and updated napp-it too to the last free one (Btw I tried to use Solaris 11.4 but couldnt get it to run in my VM so since this was not the primary focus i skipped it for now)
I dabbled around a bit and got speeds below expectations , so now want to check what the issue is.
Since my goal is to optimize for QD1T1 I am looking at single thread performance here; the final environment will have 40G networking, but for now I am Intra ESX until a second node makes sense, so should not be limited by this.
I started by analyzing networking (o/c I performed appliance tuning first) since I saw hard limits on Atto.
To identify the culprit I run two Win2012 vms and the napp IT VM (where i installed iperf2 in addition to 3 to be able to easily run multi-process tests)
Network speed between my two identical WinVMs:
Code:
C:\>iperf -c 192.168.124.28
------------------------------------------------------------
Client connecting to 192.168.124.28, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[216] local 192.168.124.32 port 49170 connected with 192.168.124.28 port 5001
[ ID] Interval Transfer Bandwidth
[216] 0.0-10.0 sec 26.3 GBytes 22.6 Gbits/sec
C:\>iperf -c 192.168.124.28 -w 1M
------------------------------------------------------------
Client connecting to 192.168.124.28, TCP port 5001
TCP window size: 1.00 MByte
------------------------------------------------------------
[216] local 192.168.124.32 port 49172 connected with 192.168.124.28 port 5001
[ ID] Interval Transfer Bandwidth
[216] 0.0-10.0 sec 25.6 GBytes 22.0 Gbits/sec
so perfectly fine for this test.
Unfortunately speed to OmniOS is way less (with MTU9000):
Napp-It Client to Win Server
Code:
~/iperf-2.0.12# /usr/local/bin/iperf -c 192.168.124.28 -w 64K -
/usr/local/bin/iperf: ignoring extra argument -- -
------------------------------------------------------------
Client connecting to 192.168.124.28, TCP port 5001
TCP window size: 64.0 KByte
------------------------------------------------------------
[ 3] local 192.168.124.30 port 55601 connected with 192.168.124.28 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 8.44 GBytes 7.25 Gbits/sec
Win Client to NappIt Server:
Code:
C:\>iperf -c 192.168.124.30
------------------------------------------------------------
Client connecting to 192.168.124.30, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[220] local 192.168.124.32 port 49216 connected with 192.168.124.30 port 5001
[ ID] Interval Transfer Bandwidth
[220] 0.0-10.0 sec 9.16 GBytes 7.86 Gbits/sec
C:\>iperf -c 192.168.124.30 -P 4
------------------------------------------------------------
Client connecting to 192.168.124.30, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[292] local 192.168.124.32 port 49220 connected with 192.168.124.30 port 5001
[268] local 192.168.124.32 port 49219 connected with 192.168.124.30 port 5001
[252] local 192.168.124.32 port 49217 connected with 192.168.124.30 port 5001
[264] local 192.168.124.32 port 49218 connected with 192.168.124.30 port 5001
[ ID] Interval Transfer Bandwidth
[292] 0.0-10.2 sec 2.70 GBytes 2.27 Gbits/sec
[268] 0.0-10.0 sec 2.39 GBytes 2.05 Gbits/sec
[252] 0.0-10.4 sec 2.78 GBytes 2.30 Gbits/sec
[264] 0.0-10.0 sec 3.37 GBytes 2.89 Gbits/sec
[SUM] 0.0-10.4 sec 11.2 GBytes 9.28 Gbits/sec
C:\>iperf -c 192.168.124.30 -w 1M
------------------------------------------------------------
Client connecting to 192.168.124.30, TCP port 5001
TCP window size: 1.00 MByte
------------------------------------------------------------
[216] local 192.168.124.32 port 49221 connected with 192.168.124.30 port 5001
[ ID] Interval Transfer Bandwidth
[216] 0.0-10.0 sec 11.8 GBytes 10.1 Gbits/sec
So with 1 M blocksize I theoretically get 10G speed, but thats far from the 22G I get from Windows. This might or might not be the cause for the below expectation speed, but its something I should work on to get the potential impact out of the way.
The question o/c is - what is the reason why OmniOS is so 'slow' ? I played around with a larger send buffer but that did not help