How to acheive SMOKIN' ZFS send/recv transfers

[K D]

Started vmotion of the vms from the host so that I can reboot it. With the current speeds, it's going to take a couple of hrs. Just realized it's 4:30 AM .

Will catch a couple of winks and check this out tomorrow after work.

 

[Rand__]

ouchie - gn8

question is - was this ever faster? What changed...

 

[K D]

The AIO was previously on a Xeon D 1518 board. Was able to get upto 400MB/s transfers then. Just set it up on the E3 board and was trying to do the initial data copy when I ran into this issue


Sent from my iPhone using Tapatalk

 

[Rand__]

interestingly my issue with slow iperf is on a E3 (v5) + ESX as well (x550 though)

edit:
Moved the ESX installation to an e5 with X540-AT2, same issue.
So not really E3 related (didnt really think so, but interesting coincidence)

 

[K D]

Booted the system off an old freenas boot disk and it shows full 10gbps speed both ways. So the issue seems to be with esxi.

 

[Rand__]

Hm try a clean install?

 

[K D]

That's the next option. Will try to do a standard install with the downloaded installer. Currently I'm using an installer created with esxi customizer integrating the x552 drivers.

 

[dicecca112]

That's the next option. Will try to do a standard install with the downloaded installer. Currently I'm using an installer created with esxi customizer integrating the x552 drivers.

Follow this

How to download and install the Intel Xeon D 10GbE X552/X557 driver/VIB for VMware ESXi 6.x, works for X540 too | TinkerTry IT @ Home

 

[K D]

Follow this

How to download and install the Intel Xeon D 10GbE X552/X557 driver/VIB for VMware ESXi 6.x, works for X540 too | TinkerTry IT @ Home

Thanks. I am aware of that option. But this is a X11SSH-CTF board with only the Intel 10gbe nics. I have to have the drivers slipstreamed into the installer to even get it to install.

 

[dicecca112]

I've actually had scenarios where the drivers work but are incredibly slow. Doing the TinkerToy fix allowed the expected performance.

 

[Rand__]

question is - was this ever faster? What changed...

The AIO was previously on a Xeon D 1518 board. Was able to get upto 400MB/s transfers then. Just set it up on the E3 board and was trying to do the initial data copy when I ran into this issue

You were using the same installation and the same ConnectX2 card? Just different board?

 

[Terry Kennedy]

You of course can ramp up the iperf with larger windows sizes ( -w 1M) or multiple threads (-P 4/8) but without knowing the exact details of the helper program that does the later test its always a bit optimistic and just proves theoretical capability

In my experience, if you don't get wire speed between neighboring machines with the default window size and single thread, something is wrong.

 

[K D]

I have tried by loading esxi 6.0 and 6.5 directly downloaded from the vmware site on to the host t1stor. Also swapped out the old Connect X2 with a Connect X3. Repeated the tests and I get the same outcome.

 

[Rand__]

In my experience, if you don't get wire speed between neighboring machines with the default window size and single thread, something is wrong.

I tend to agree.
Ramping up tells you only if it's a wire/network problem or not.

Question is how to troubleshoot efficiently.

Basically you need to change everything in the stack top down or bottom up (or most likely).


So in this case - different cable/switch ports?

 

[K D]

I have tried different cables, NIC, switch ports and even tried with a Cisco SG550 switch with no change in the outcome.

Due to making some upgrades I could not test with a different system altogether. Parking it for this week as it is. I'll bring up new ESXI and freenas systems on different boxes over the weekend and repeat the tests.

Meanwhile anyone coming across this thread please let me know if there is something else I can try or any data/logs that will help troubleshoot.

 

[K D]

I managed to bring online another esxi host connected to the same switch and ran the tests. I'm able to get near wire speeds.




From the FreeNAS box to both esxi servers



Looks like the issue is with the FreeNAS box and not esxi.

 

[K D]

Here is the ifconfig result from the freenas box

mlxen0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
capabilities=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:02:c9:57:16:12
inet 172.16.10.115 netmask 0xffffff00 broadcast 172.16.10.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
status: active
supported media:
media autoselect
media 40Gbase-CR4 mediaopt full-duplex
media 10Gbase-CX4 mediaopt full-duplex
media 10Gbase-SR mediaopt full-duplex
media 1000baseT mediaopt full-duplex

 

[marcoi]

shouldnt you have MTU set to 9000?

 

[K D]

shouldnt you have MTU set to 9000?

I have not currently enabled jumbo frames. But would that account for a 95% difference in speed?

 

[marcoi]

probably not, especially since you do not have jumbo frames enabled. Have you tried moving the nic to a different pci slot?