Slow vmotion in ESX (nvme2nvme@56GbE)

[Rand__]

So,
since I am unhappy with vsan atm I am playing around with the hardware to establish a baseline what it could do.
Currently I am running a few simple tests moving test VMs from one box to another via vMotion to test the potential speed.
I have various NVMe drives in both my boxes and a 56GbE link between them (switched).

1. If i move VMs from one drive to another on the same host I get speeds from 700MB/s to 1G/s (on both hosts)
2. Iperf between the two hosts is ok (4K MTU)
13.8 Gbits/sec (64K Window size), 21.6 Gbits/sec (1M Window size), single thread (A->B)
15.9 Gbits/sec (64K Window size), 19.8 Gbits/sec (1M Window size), single thread (B->A)
3. Transfer speeds between the two hosts is not so good - between 180MB/s and 260 MB/s - the same drives that push tripple that locally.

So -
is that to be expected?
Anything I could check?

I have the most recent nvme driver, most of the drives have the current fw.
Mellanox cards are using the ESX 6.5 driver o/c due to the lack of alternatives

CPU Utilization seems fine, have not seen a single core over 40% ...

Will test more with a third host, but am kind of lost (again)...

 

[Evan]

What's your network config ? Are management and vSAN different vswitch and subnet ?
Found issue where I assume the vMotion traffic is throttled when in same vlan of management.

 

[Rand__]

Yes, management is on a separate 1GBe connection, vsan and vmotion have a dedicated vswitch and one vmk each:

Iperf was run on the vsan vmk since I have not found yet how to make iperf use the vmotion stack (found it for vmkping which is fine).

Question - is iperf a good enough indicator that the speed between the nodes is sufficient?
Guess I could do a RAMdisk 2 RamDisk transfer to test

Edit:
So I moved my vsan Interface around and ran a Win2Win RamDisk2RamDisk test - not great but better than vmWare - getting speeds around 600 MB/s (vmxnet3, no further optimization)

 

[Rand__]

I adjusted the MTU to 9k (4k was IB limit i think) and ran a couple of parallel vMotions - 450 MB/s ... better but not great.