VMWare network routing using unassigned NICs?

[Rand__]

Hi,

I am setting up a test scenario for Chelsio vs MLX Nic (RDMA) tests.
I have a ESXi7 box with a MLX and a Chelsio adapter, assigned to a dvswitch with 4 portgroups (VLANs 10/11 {Uplink1/2}, 20/21 {Uplink3/4}).


I have mounted a NFS share on a TNC 12 box also with a Chelsio and MLX adapter, providing a couple of NFS shares on 4 IPs ( vlan 10/11 for mlx, vlan 20/21 for the chelsio).

I then mounted the nfs shares as follows:

esxcli storage nfs41 add -H 10.0.10.49,10.0.11.49 -s /mnt/sas/fn12_mlx_async_64k -v fn12_mlx_async_64k
esxcli storage nfs41 add -H 10.0.10.49,10.0.11.49 -s /mnt/sas/fn12_mlx_sync_64k -v fn12_mlx_sync_64k

esxcli storage nfs41 add -H 10.0.20.49,10.0.21.49 -s /mnt/sas/fn12_chelsio_async_64k -v fn12_chelsio_async_64k
esxcli storage nfs41 add -H 10.0.20.49,10.0.21.49 -s /mnt/sas/fn12_chelsio_sync_64k -v fn12_chelsio_sync_64k

Theoretically that should bind each NFS based datastore to the vmks in that ip range which is bound to a portgroup with the vlan which is bound to an uplink which is implemented by either the Chelsio or MLX card.

I then moved a testvm on that datastore and ran fio on it.

Now what happens is that on whichever datastore I run a fio test I see that all 4 NIC ports on the TNC port receive traffic and i dont understand why.
My only idea so far is that it might have to do with the Load Balancing option in the dvswitch but that *should* honor unused nics (theoretically)...

I tried various options and at this point at this one, but neither of them helped at all.



I am confused to say the least

I'll try and deny access to the MLX sharrres to chelsio ips in TNC (and vice versa) but it should work without that patch imho...

 

[Rand__]

Something is muxing those connections ... when i block the ips on TNC side I get access issues... weird.

I'll fall back to turn off ports for now...

 

[Rand__]

Even if I turn off the Chelsio ports it happily connects via the MLX ports - that o/c since both nics are trunked on all vlans, but the main issue is that it totally ignores the assigned Uplinks here.

It probably would work if i deactivated the Chelsio on TNC side, but I've now decided to move the chelsio to a second dvs instead, that is the cleaner solution i guess

I mean its nice to distribute the load over 4 ports, but I am not quite reaching 400Gbit on my filer to really make use of that...

 

[Rand__]

I begin to think that ESXi is not happy mixing MLX (roce) and Chelsio (iWarp) based rdma...
When trying to 'discourage' use of the MLX cards the Chelsio's don't work properly any more...

After a reboot i can mount a Chelsio mapped nfs export to a datastore just fine, 5 seconds later its unavailable (despite me being able to ping the box just fine)...

Log shows thing like

syslog.log:2022-04-23T16:04:25.242Z dcbd[2100877]: [error]    set_ieee_hw_ets: Invalid operating mode for device vmnic3
syslog.log:2022-04-23T16:04:25.242Z dcbd[2100877]: [error]    Message ID(18) MSG_ERR_PFC_NONOPER: {0c 42 a1 51 35 c1}
syslog.log:2022-04-23T16:04:25.242Z dcbd[2100877]: [error]    set_ieee_hw_pfc: Invalid operating mode for device vmnic3
syslog.log:2022-04-23T16:04:25.242Z dcbd[2100877]: [error]    Message ID(2) MSG_ERR_APP_NONOPER: {0c 42 a1 51 35 c1} (RoCEv2)
syslog.log:2022-04-23T16:04:25.242Z dcbd[2100877]: [error]    set_ieee_hw_app: Failed status(195887136)
syslog.log:2022-04-23T16:04:25.242Z dcbd[2100877]: [error]    set_ieee_hw_app: Failed.
syslog.log:2022-04-23T16:04:25.242Z dcbd[2100877]: [error]    set_dcb_cfg: Failed status(195887235)

which is o/c incorrect since Chelsio's dont use/need DCB...

Or its the roce enabled MLX switch... who knows.

I think I Will move to one Chelsio box and one MLX box for now to do the TrueNas tests before going back testing vsan over rdma