YEAHHH BUDDDYYY ... that's the money shot right there! ESXI <=> ESXi via direct 40G = iperf shows 33GHave you run the command with multiple threads (ie -P 4) (if its iperf2 - iperf 3 needs multiple instances) and larger window size -w 1M| -w 8M ?
This should show you the theoretical maximum
Curiously though I can't get above 14G FreeNAS <=> FreeNAS on those same hosts.
I picked up ~+1G on the 10G port (FreeNAS to FreeNAS), so I'm going to assume that was the result of properly enabling jumbo frames. mtu was set to 9000 in ESXi and FreeNAS, globally enabled on the ICX-6450 (i.e. "jumbo"), BUT NOT on the vlan interface. The following command did the trick and now a Plex jail works again (yay for not getting yelled at any longer).
Code:
interface ve200
ip mtu 9000
Code:
[root@ESXi-01:/opt/iperf/bin] ./iperf -c 10.2.0.42 -w 1M -P 8
------------------------------------------------------------
Client connecting to 10.2.0.42, TCP port 5001
TCP window size: 1.01 MByte (WARNING: requested 1.00 MByte)
------------------------------------------------------------
[ 10] local 10.2.0.41 port 51426 connected with 10.2.0.42 port 5001
[ 8] local 10.2.0.41 port 21429 connected with 10.2.0.42 port 5001
[ 7] local 10.2.0.41 port 28275 connected with 10.2.0.42 port 5001
[ 6] local 10.2.0.41 port 10155 connected with 10.2.0.42 port 5001
[ 11] local 10.2.0.41 port 48648 connected with 10.2.0.42 port 5001
[ 5] local 10.2.0.41 port 62808 connected with 10.2.0.42 port 5001
[ 4] local 10.2.0.41 port 19689 connected with 10.2.0.42 port 5001
[ 3] local 10.2.0.41 port 57074 connected with 10.2.0.42 port 5001
[ ID] Interval Transfer Bandwidth
[ 10] 0.0-10.0 sec 4.82 GBytes 4.14 Gbits/sec
[ 8] 0.0-10.0 sec 4.80 GBytes 4.13 Gbits/sec
[ 7] 0.0-10.0 sec 4.85 GBytes 4.17 Gbits/sec
[ 6] 0.0-10.0 sec 4.83 GBytes 4.15 Gbits/sec
[ 11] 0.0-10.0 sec 4.81 GBytes 4.14 Gbits/sec
[ 5] 0.0-10.0 sec 4.81 GBytes 4.13 Gbits/sec
[ 4] 0.0-10.0 sec 4.87 GBytes 4.18 Gbits/sec
[ 3] 0.0-10.0 sec 4.85 GBytes 4.17 Gbits/sec
[SUM] 0.0-10.0 sec 38.6 GBytes 33.2 Gbits/sec