Anyone able to run Roce v2 on CX-3 VPI in ESXi 7.0?

vangoose

Active Member
May 21, 2019
263
69
28
Canada
Ran all the required commands to modify kernel param

Code:
esxcli system module parameters set -m nmlx4_en -p "pfctx=0x08 pfcrx=0x08"
esxcli system module parameters set -m nmlx4_rdma -p "dscp_force=26"
esxcli system module parameters set -m nmlx4_core -p "enable_rocev2=1"


esxcli system module parameters list -m nmlx4_core
Name                    Type  Value  Description
----------------------  ----  -----  -----------
enable_64b_cqe_eqe      int          Enable 64 byte CQEs/EQEs when the the FW supports this
   Values : 1 - enabled, 0 - disabled
   Default: 0
enable_dmfs             int          Enable Device Managed Flow Steering
   Values : 1 - enabled, 0 - disabled
   Default: 1
enable_qos              int          Enable Quality of Service support in the HCA
   Values : 1 - enabled, 0 - disabled
   Default: 0
enable_rocev2           int   1      Enable RoCEv2 mode for all devices
   Values : 1 - enabled, 0 - disabled
   Default: 0
Code:
esxcli rdma device list
Name     Driver      State    MTU  Speed    Paired Uplink  Description
-------  ----------  ------  ----  -------  -------------  -----------
vmrdma0  nmlx4_rdma  Active  1024  40 Gbps  vmnic0         MT27500 Family [ConnectX-3]
vmrdma1  nmlx4_rdma  Active  1024  40 Gbps  vmnic1000002   MT27500 Family [ConnectX-3]
vmrdma2  nmlx5_rdma  Down    1024  0        vmnic1         MT27710 Family  [ConnectX-4 Lx]
vmrdma3  nmlx5_rdma  Down    1024  0        vmnic2         MT27710 Family  [ConnectX-4 Lx]


esxcli rdma device protocol list
Device   RoCE v1  RoCE v2  iWARP
-------  -------  -------  -----
vmrdma0     true    false  false
vmrdma1     true    false  false
vmrdma2     true     true  false
vmrdma3     true     true  false
Still show as not enabled. I guess it's showing its ages now with ESXi 7.
 

i386

Well-Known Member
Mar 18, 2016
1,992
516
113
31
Germany
Connectx-3 support RoCE v1
Connectx-3 Pro and newer support RoCE v2 (and v1)

Cx-3 pro uses slightly newer asic than the normal cx-3.
 

i386

Well-Known Member
Mar 18, 2016
1,992
516
113
31
Germany

vangoose

Active Member
May 21, 2019
263
69
28
Canada
Nope, VPI (Virtual protocoll Interconnect) is the the term/technology mellanox uses for the adapters that support ethernet and infiniband. (https://www.mellanox.com/pdf/prod_architecture/Virtual_Protocol_Interconnect_VPI.pdf)

About RoCE support on the different adapters:

View attachment 15154
Source:
Page 114
Thanks, this is an excellent doc.