How to use the mlx5_core driver with Mellanox ConnectX-4 Lx in Debian?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

prdtabim

Active Member
Jan 29, 2022
170
66
28
Update I can't get these cards to connect to each other directly at anything over 1Gbps. I'm giving up on this lol I've lost too much time fiddling with these cards / switch.
If is connects at 1Gb/s but not at 10Gb/s implies that your configuration is OK but you may have a cable defect , a cable incompatibility or a damaged port in one of the cards. I buy come DAC/AOC cables from ebay and one of them have this problem.
 

crackelf

Member
Apr 11, 2021
74
6
8
If is connects at 1Gb/s but not at 10Gb/s implies that your configuration is OK but you may have a cable defect , a cable incompatibility or a damaged port in one of the cards. I buy come DAC/AOC cables from ebay and one of them have this problem.
I've ordered a few different OS2 cables / DAC's to test this as a last ditch solution. Also got some different ConnectX-3's to see if the cards are screwy. That feels right that the cables are the culprit, but I've given up trying to intuit which thing is going wrong here haha. Everything arrives end of the week will update this thread as it goes!
 
Last edited:

crackelf

Member
Apr 11, 2021
74
6
8
If is connects at 1Gb/s but not at 10Gb/s implies that your configuration is OK but you may have a cable defect , a cable incompatibility or a damaged port in one of the cards. I buy come DAC/AOC cables from ebay and one of them have this problem.
Just to clarify my setup here in case anything looks wrong:
OS2 LC-LC Singlemode Duplex
KAIAM QSFP+40G-LR4 lite
Mellanox MCX4131A-GCAT_C05
 

llowrey

Active Member
Feb 26, 2018
167
138
43
LX4 optics are hi-power, relative to DACs or SR4 and, at least in the case of the Mellanox SX60xx series of switch, are not well supported. I struggled for years to get one to work with my SX6018, hacking away at the firmware to no avail... then one day a good samaritan PM'd me a secret cli command to enable LX4 and I was off to the races.

Unless you need duplex LC/LC, I would recommend getting a cheap SR4 transceiver and a cheap OM4 MPT cable. Or, if your equipment is close enough and bend radii aren't an issue, just go with a DAC.
 
  • Like
Reactions: crackelf

crackelf

Member
Apr 11, 2021
74
6
8
LX4 optics are hi-power, relative to DACs or SR4 and, at least in the case of the Mellanox SX60xx series of switch, are not well supported. I struggled for years to get one to work with my SX6018, hacking away at the firmware to no avail... then one day a good samaritan PM'd me a secret cli command to enable LX4 and I was off to the races.

Unless you need duplex LC/LC, I would recommend getting a cheap SR4 transceiver and a cheap OM4 MPT cable. Or, if your equipment is close enough and bend radii aren't an issue, just go with a DAC.
That's funny... that was *exactly* my original plan:
Thanks for the response and the link! I had been looking at these Brocade optics on eBay and was going to try out these Arista MTP-MTP 12 fibers OM4 (also eBay)
but was convinced otherwise
yes they're regular 40gbe ports. instead of expensive annoying mtp you can grab these BiDi optics and run 40gbE over cheap regular singlemode duplex LC fiber XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS | eBay
Thank you for the info. I've ordered some DAC's, SR4, and OM4.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
LX4 optics are hi-power, relative to DACs or SR4 and, at least in the case of the Mellanox SX60xx series of switch, are not well supported. I struggled for years to get one to work with my SX6018, hacking away at the firmware to no avail... then one day a good samaritan PM'd me a secret cli command to enable LX4 and I was off to the races.

Unless you need duplex LC/LC, I would recommend getting a cheap SR4 transceiver and a cheap OM4 MPT cable. Or, if your equipment is close enough and bend radii aren't an issue, just go with a DAC.
they're not LR4, they're LR4 Lite - they use the same power as most SR4 modules and even less than some I've used. Quite a few of us in the brocade thread are using those KAIAM modules in Mellanox's with zero issue. Wouldn't have recommended them (and saved the annoying purchase of MTP) otherwise
 
Last edited:
  • Like
Reactions: llowrey and Stephan

NablaSquaredG

Layer 1 Magician
Aug 17, 2020
1,319
800
113
then one day a good samaritan PM'd me a secret cli command to enable LX4 and I was off to the races.
Would you mind sharing this CLI command? I have also had a fair amount of issues with SX6036 + LR4 Transceivers in the last couple of days (result: only Mellanox branded / programmed LR4 optics seem to be accepted by the SX6036)

Having that said, LR4 optics + ConnectX-5 are also a bit problematic. In Infiniband mode, the card only links up to SDR (10Gbit)...

they're not LR4, they're LR4 Lite - they use the same power as most SR4 modules and even less than some I've used. Quite a few of us in the brocade thread are using those KAIAM modules in Mellanox's with zero issue. Wouldn't have recommended them (and saved the annoying purchase of MTP) otherwise
Mellanox switches or cards?

In cards in Ethernet everything works fine, but in SX6036 switches I have tried Finisar FTL4C1QL2L LR4-Lite and FS.com modules and both only worked after I patched the vendor to "Mellanox"
Even worse ConnectX-5 and Infiniband, because the CX-5 seems to do a deeper inspection and still detects the modules as "FS" whereas the switch happily reports "Mellanox"
 
Last edited:
  • Like
Reactions: crackelf

crackelf

Member
Apr 11, 2021
74
6
8
Would you mind sharing this CLI command? I have also had a fair amount of issues with SX6036 + LR4 Transceivers in the last couple of days (result: only Mellanox branded / programmed LR4 optics seem to be accepted by the SX6036)

Having that said, LR4 optics + ConnectX-5 are also a bit problematic. In Infiniband mode, the card only links up to SDR (10Gbit)...


Mellanox switches or cards?

In cards in Ethernet everything works fine, but in SX6036 switches I have tried Finisar FTL4C1QL2L LR4-Lite and FS.com modules and both only worked after I patched the vendor to "Mellanox"
Even worse ConnectX-5 and Infiniband, because the CX-5 seems to do a deeper inspection and still detects the modules as "FS" whereas the switch happily reports "Mellanox"
This has me thinking I need specific modules too. My cards link out to the X-5 page even though they're allegedly 4's... Reassuring someone else is having issues with this too. Thanks for the report! I have some Mellanox DACs coming soon to test.
 

crackelf

Member
Apr 11, 2021
74
6
8
Looping in this post from @i386
I was hoping to see something like hpe typcal product numbers eg "123456-B85". But the numbers look like "vanilla" mellanox nics which is confusing becuase the sellers says that the psid was changed (which usually is done to be able to flash stock mellanox firmware over oem firmware)

Another idea or possible cause: revision a1 + "made in Israel" could be an earlier version (or engineering sample). All the connectx-4 nics that I have are a2 or newer and the hpe branded models are all "made in China", the vanilla cx-4 are all "made in Israel"
Hopefully by the end of this week or middle of next I'll have answers for what exactly is happening here.
 

crackelf

Member
Apr 11, 2021
74
6
8
Got some X3 VPI cards in
Code:
Configurations:                              Next Boot       New
         LINK_TYPE_P1                        N/A             ETH(2)       

Apply new Configuration? (y/n) [n] : y
Applying... Failed!
-E- Failed to set configuration: Failed to get default parameters: Operation not supported please specify all the parameters for VPI settings.
echo eth > /sys/devices/pcixxxx/xx/mlx4_port1 enables the eth, but I still can't run the param change
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
Got some X3 VPI cards in
Code:
Configurations:                              Next Boot       New
         LINK_TYPE_P1                        N/A             ETH(2)      

Apply new Configuration? (y/n) [n] : y
Applying... Failed!
-E- Failed to set configuration: Failed to get default parameters: Operation not supported please specify all the parameters for VPI settings.
echo eth > /sys/devices/pcixxxx/xx/mlx4_port1 enables the eth, but I still can't run the param change
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

 
  • Like
Reactions: crackelf

crackelf

Member
Apr 11, 2021
74
6
8
I verify and sysfs is read only in the mlx4_core module.
To configure the card you will pass module params or in some cases use mlxconfig .
Yes in the end I went the module route for part of it, sysfs for the eth enablement, and the mellanox config tool for SRIOV. Strange combination.
/etc/modprobe.d/mlx4_core.conf looks like
Code:
options mlx4_core num_vfs=2 port_type_array=2 probe_vf=2
mstconfig -d set SRIOV_EN=1 NUM_OF_VFS=2
and
echo eth > /sys/devices/pcixxxx/xx/mlx4_port1
 

crackelf

Member
Apr 11, 2021
74
6
8
Still not able to get this to stick, but thanks for the link! Somehow I've gotten SRIOV working and ethernet enabled despite the firmware disagreeing. See quote. Will try flashing / updating firmware to see if that changes anything. See you in the other thread ;)
I verify and sysfs is read only in the mlx4_core module.
To configure the card you will pass module params or in some cases use mlxconfig .
Yes in the end I went the module route for part of it, sysfs for the eth enablement, and the mellanox config tool for SRIOV. Strange combination.
/etc/modprobe.d/mlx4_core.conf looks like
Code:
options mlx4_core num_vfs=2 port_type_array=2 probe_vf=2
mstconfig -d set SRIOV_EN=1 NUM_OF_VFS=2
and
echo eth > /sys/devices/pcixxxx/xx/mlx4_port1
 
Last edited:

crackelf

Member
Apr 11, 2021
74
6
8
(result: only Mellanox branded / programmed LR4 optics seem to be accepted by the SX6036)
You ended up being right! Mellanox transceivers are all I've been able to get working with these.

I'll make a support matrix next week after some testing and post it here. Thank you for the insight. I would never have guessed.