7050QX-32S DCB (Lossless ethernet)

[dante4]

get ESXi to play nice with "older" hardware

Welp. 7.0 plays nice with ConnectX-3 Pro, in both terms of drivers and firmware tools. Since well, they are in HCL.

Ubuntu is not really, since you can't use non-inbox drivers on newer kernel

 

[dante4]

@dante4

Just request help from ESXi support and come back considering apologising for being rude to the guys.

If I would have platinum ESXi partner that would be solution indeed. But in such case there would be no need to use ConnectX-3 cards in first place, right?

 

[dante4]

CX4s are PCIe gen 3 and the maximum bandwidth available is only 126Gb. Having 2 ports will only allow you to have failover not use both of them at the same time.

This thread reminds me of

• Good
• Fast
• Cheap

Pick any 2.

Erm, you clearly messed up your math. Or you are talking about CX3. CX3 is pcie 3.0 x8 cards. So max bandwidth for NIC will be 7880 MB/s, i.e. 63 040 Mbits/s, or around 1.5 ports under full load (outside of topic that's why right now I'm also buying one more CX3 to make storage sever connected by 4 ports to eliminate possible bottleneck of this, MPIO is such nice thing to have)

CX4 is x16, that's 126 080 Mbits/s, depending on model it can be PCie 3.0 or 4.0. For 4.0 it's enough for active-active 100G, but 100G switch a bit too costly for homelab)


That's still out of topic question, regarding DCBx. Since everyone starts to talk about *Just use ConnectX-4* and continued to discuss unrelated matter which is as helpful to topic as silence, but hey. Maybe someone who actually know how to do DCBx in correct way will stumble under such active thread, since I'm not even sure that I have correctly enabled it on Ubuntu, because while I do see Pause frame when I simulated overheat of NIC (so it throttle down speed) - there is still "No application priority configuration TLV received", while it should be enabled from software

 

[dante4]

I have several pro and non pro, both has random issues,

Like which? Outside of RDMA vSAN lab (which is unsurprised, since they are not in HCL) I had no problem with them, so I'm genuinely interested what kind of problem you have met with Pro version

 

[dante4]

If telling you "a car won't fly like a plane" is considered "unhelpful"

Not really correct comparison. It would be correct if I was asking something like "How to reach 80G from single CX3 pro", which is technically impossible.

Correct comparison is "an average car won't speed up to 200 km/h", and indeed without tweaks to itself it will not reach such speed

 

[dante4]

So results as of today, would be helpful to receive the correction to config, since I'm not expert in DCBx

Switch (DCS-7050QX-32S 4.28.10.1M) config:

Spoiler: Global

dcbx application udp 4791 priority 3
interface defaults
mtu 9000
platform trident mmu queue profile RoCELosslessProfile
ingress threshold 1/16
egress unicast queue 3 threshold 8

Spoiler: Interface specific config:

dcbx mode ieee
speed forced 40gfull
qos trust dscp
priority-flow-control on
priority-flow-control priority 3 no-drop
!
tx-queue 3
bandwidth guaranteed 20000000
random-detect ecn minimum-threshold 150 kbytes maximum-threshold 1500 kbytes max-mark-probability 100 weight 0

ESXi config:

esxcli system module parameters set -m nmlx4_en -p "pfctx=0x08 pfcrx=0x08"
esxcli system module parameters set -m nmlx4_rdma -p "dscp_force=26 pcp_force=3"
esxcli system module parameters set -m nmlx4_core -p "enable_qos=1 enable_rocev2=1"

Ubuntu config:

NVIDIA Firmware Tools (MFT)

The NVIDIA Firmware Tools (MFT) package is a set of firmware management tools

network.nvidia.com

mft-4.22.1-406-x86_64-deb.tgz - enables config + temperature monitoring

(ens160 is CX3 pro port, needs to be repeated for second port)

dcbtool sc ens160 pfc pfcup:00001000

lldptool -t -i ens160 -V PFC -c enabled
lldptool -T -i ens160 -V PFC enabled=3

lldptool set-lldp -i ens160 adminStatus=rxtx ;
lldptool -T -i ens160 -V sysName enableTx=yes;
lldptool -T -i ens160 -V portDesc enableTx=yes ;
lldptool -T -i ens160 -V sysDesc enableTx=yes
lldptool -T -i ens160 -V sysCap enableTx=yess
lldptool -T -i ens160 -V mngAddr enableTx=yess
lldptool -T -i ens160 -V PFC enableTx=yes;
lldptool -T -I ens160 -V CEE-DCBX enableTx=yes;

mkdir /sys/kernel/config/rdma_cm/mlx4_0/ports/
will allow you to check default mode, it should be RoCE v2
Maybe default ToS should be 96, based on this, but I'm unsure.

Results:

Spoiler: show dcbx

For both ESXi - I have this, but that is because I have enabled "dcbx mode ieee" on this port

IEEE DCBX is enabled and active
Last LLDPDU received on Wed Apr 24 12:04:01 2024
No priority flow control configuration TLV received
No application priority configuration TLV received

Ethernet18/1: - ubuntu, port 1
IEEE DCBX is enabled and active
Last LLDPDU received on Sat May 18 13:55:03 2024
- PFC configuration: willing
capable of bypassing MACsec
supports PFC on up to 8 traffic classes
PFC enabled on priorities: 3
No application priority configuration TLV received

Ethernet24/1: - ubuntu, port 2
IEEE DCBX is enabled and active
Last LLDPDU received on Sat May 18 13:32:31 2024
- PFC configuration: willing
capable of bypassing MACsec
supports PFC on up to 8 traffic classes
PFC enabled on priorities: 3
No application priority configuration TLV received

 

[klui]

Erm, you clearly messed up your math. Or you are talking about CX3. CX3 is pcie 3.0 x8 cards. So max bandwidth for NIC will be 7880 MB/s, i.e. 63 040 Mbits/s, or around 1.5 ports under full load (outside of topic that's why right now I'm also buying one more CX3 to make storage sever connected by 4 ports to eliminate possible bottleneck of this, MPIO is such nice thing to have)

CX4 is x16, that's 126 080 Mbits/s, depending on model it can be PCie 3.0 or 4.0. For 4.0 it's enough for active-active 100G, but 100G switch a bit too costly for homelab)

I advise you to look at ConnectX-4 specifications again. Specifications

Mellanox didn't support PCIe Gen 4 until ConnectX-5. Specifications

For your reference, PCIe gen 3 x16 is 15.754 GB/s PCI Express - Wikipedia

 

[dante4]

I advise you to look at ConnectX-4 specifications again. Specifications

Mellanox didn't support PCIe Gen 4 until ConnectX-5. Specifications

Indeed, you are correct. Missed that CX4 didn't had 4.0.

For your reference, PCIe gen 3 x16 is 15.754 GB/s PCI Express - Wikipedia

Erm. And? Which point of mine it counters? x1 pcie 3.0 line is 985 MB/s. 985*16 is 15760 MB/s. Which is 126080 Mb/s.