Dabbling with RDMA ...
Test ESXi boxes (6.7U3) have CX3's (non pro so RoCE v1 only)
followed https://community.mellanox.com/s/article/howto-run-roce-over-l2-enabled-with-pfc--esxi-x
to enable on ESXi
Basically on each I did
esxcli system module parameters set -m nmlx4_en -p "pfctx=0x08 pfcrx=0x08"
esxcli system module parameters set -m nmlx4_rdma -p "pcp_force=3"
reboot
On the switch (6012 in ETH) I followed
https://community.mellanox.com/s/article/howto-enable-pfc-on-mellanox-switches--switchx-x
interface ethernet 1/1-1/6 flowcontrol send off force
interface ethernet 1/1-1/6 flowcontrol receive off force
dcb priority-flow-control enable
dcb priority-flow-control priority 3 enable
interface ethernet 1/1-1/6 dcb priority-flow-control mode on force
I didnt modify my current VLAN settings as I have a bunch set up for different ESXi traffic already.
On ESXi there was no mapping mentioned of VLAN to priority class (on Linux they map it with vconfig per VLAN -> https://community.mellanox.com/s/article/howto-run-roce-and-tcp-over-l2-enabled-with-pfc), so I didnt do anything else.
Networking is still working fine, no errors, but no RoCE seems to be in use - I can't see a single pause frame...
So anyone any idea what I am missing?
Edit:
Could it be that regular ESXi traffic will not generate RDMA traffice?
I had hoped from the description
"
RDMA in ESXi
that it might run in native ESXi too when enabled on interface level.
Only thing I tested till now were some vMotions...
Test ESXi boxes (6.7U3) have CX3's (non pro so RoCE v1 only)
followed https://community.mellanox.com/s/article/howto-run-roce-over-l2-enabled-with-pfc--esxi-x
to enable on ESXi
Basically on each I did
esxcli system module parameters set -m nmlx4_en -p "pfctx=0x08 pfcrx=0x08"
esxcli system module parameters set -m nmlx4_rdma -p "pcp_force=3"
reboot
On the switch (6012 in ETH) I followed
https://community.mellanox.com/s/article/howto-enable-pfc-on-mellanox-switches--switchx-x
interface ethernet 1/1-1/6 flowcontrol send off force
interface ethernet 1/1-1/6 flowcontrol receive off force
dcb priority-flow-control enable
dcb priority-flow-control priority 3 enable
interface ethernet 1/1-1/6 dcb priority-flow-control mode on force
I didnt modify my current VLAN settings as I have a bunch set up for different ESXi traffic already.
On ESXi there was no mapping mentioned of VLAN to priority class (on Linux they map it with vconfig per VLAN -> https://community.mellanox.com/s/article/howto-run-roce-and-tcp-over-l2-enabled-with-pfc), so I didnt do anything else.
Networking is still working fine, no errors, but no RoCE seems to be in use - I can't see a single pause frame...
So anyone any idea what I am missing?
Edit:
Could it be that regular ESXi traffic will not generate RDMA traffice?
I had hoped from the description
"
RDMA in ESXi
- RDMA can be useful to accelerate many of the hypervisor services including; SMP-FT, NFS and iSCSI.
that it might run in native ESXi too when enabled on interface level.
Only thing I tested till now were some vMotions...
Last edited: