How to use the mlx5_core driver with Mellanox ConnectX-4 Lx in Debian?

[prdtabim]

Update I can't get these cards to connect to each other directly at anything over 1Gbps. I'm giving up on this lol I've lost too much time fiddling with these cards / switch.

If is connects at 1Gb/s but not at 10Gb/s implies that your configuration is OK but you may have a cable defect , a cable incompatibility or a damaged port in one of the cards. I buy come DAC/AOC cables from ebay and one of them have this problem.

 

[crackelf]

If is connects at 1Gb/s but not at 10Gb/s implies that your configuration is OK but you may have a cable defect , a cable incompatibility or a damaged port in one of the cards. I buy come DAC/AOC cables from ebay and one of them have this problem.

I've ordered a few different OS2 cables / DAC's to test this as a last ditch solution. Also got some different ConnectX-3's to see if the cards are screwy. That feels right that the cables are the culprit, but I've given up trying to intuit which thing is going wrong here haha. Everything arrives end of the week will update this thread as it goes!

 

[crackelf]

If is connects at 1Gb/s but not at 10Gb/s implies that your configuration is OK but you may have a cable defect , a cable incompatibility or a damaged port in one of the cards. I buy come DAC/AOC cables from ebay and one of them have this problem.

Just to clarify my setup here in case anything looks wrong:
OS2 LC-LC Singlemode Duplex

OS2 9/125 LC-LC Singlemode Fiber Optic Cable Duplex 7m(23ft) - Yellow - FS.com

Buy 7m (23ft) OS2 9/125 LC-LC Singlemode Fiber Patch Cable Duplex 2.0mm PVC(OFNR) at reliable fiber Jumper manufacturer FS.COM. In stock, Custom Service on-line.

www.fs.com

KAIAM QSFP+40G-LR4 lite

XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS | eBay

Find many great new & used options and get the best deals for XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Mellanox MCX4131A-GCAT_C05

Mellanox MCX4131A-GCAT_C05 ConnectX-4 LX 50GbE PCIe Network Card Newest Firmware | eBay

Find many great new & used options and get the best deals for Mellanox MCX4131A-GCAT_C05 ConnectX-4 LX 50GbE PCIe Network Card Newest Firmware at the best online prices at eBay! Free shipping for many products!

www.ebay.com

 

[llowrey]

LX4 optics are hi-power, relative to DACs or SR4 and, at least in the case of the Mellanox SX60xx series of switch, are not well supported. I struggled for years to get one to work with my SX6018, hacking away at the firmware to no avail... then one day a good samaritan PM'd me a secret cli command to enable LX4 and I was off to the races.

Unless you need duplex LC/LC, I would recommend getting a cheap SR4 transceiver and a cheap OM4 MPT cable. Or, if your equipment is close enough and bend radii aren't an issue, just go with a DAC.

 

[Stephan]

What secret cli command?

 

[llowrey]

What secret cli command?

PM'd

 

[crackelf]

LX4 optics are hi-power, relative to DACs or SR4 and, at least in the case of the Mellanox SX60xx series of switch, are not well supported. I struggled for years to get one to work with my SX6018, hacking away at the firmware to no avail... then one day a good samaritan PM'd me a secret cli command to enable LX4 and I was off to the races.

Unless you need duplex LC/LC, I would recommend getting a cheap SR4 transceiver and a cheap OM4 MPT cable. Or, if your equipment is close enough and bend radii aren't an issue, just go with a DAC.

That's funny... that was *exactly* my original plan:

Thanks for the response and the link! I had been looking at these Brocade optics on eBay and was going to try out these Arista MTP-MTP 12 fibers OM4 (also eBay)

but was convinced otherwise

yes they're regular 40gbe ports. instead of expensive annoying mtp you can grab these BiDi optics and run 40gbE over cheap regular singlemode duplex LC fiber XQX2502 KAIAM QSFP+40G-LR4 Lite OPTICAL MODULE NEW PULLS | eBay

Thank you for the info. I've ordered some DAC's, SR4, and OM4.

 

[fohdeesha]

LX4 optics are hi-power, relative to DACs or SR4 and, at least in the case of the Mellanox SX60xx series of switch, are not well supported. I struggled for years to get one to work with my SX6018, hacking away at the firmware to no avail... then one day a good samaritan PM'd me a secret cli command to enable LX4 and I was off to the races.

Unless you need duplex LC/LC, I would recommend getting a cheap SR4 transceiver and a cheap OM4 MPT cable. Or, if your equipment is close enough and bend radii aren't an issue, just go with a DAC.

they're not LR4, they're LR4 Lite - they use the same power as most SR4 modules and even less than some I've used. Quite a few of us in the brocade thread are using those KAIAM modules in Mellanox's with zero issue. Wouldn't have recommended them (and saved the annoying purchase of MTP) otherwise

 

[NablaSquaredG]

then one day a good samaritan PM'd me a secret cli command to enable LX4 and I was off to the races.

Would you mind sharing this CLI command? I have also had a fair amount of issues with SX6036 + LR4 Transceivers in the last couple of days (result: only Mellanox branded / programmed LR4 optics seem to be accepted by the SX6036)

Having that said, LR4 optics + ConnectX-5 are also a bit problematic. In Infiniband mode, the card only links up to SDR (10Gbit)...

they're not LR4, they're LR4 Lite - they use the same power as most SR4 modules and even less than some I've used. Quite a few of us in the brocade thread are using those KAIAM modules in Mellanox's with zero issue. Wouldn't have recommended them (and saved the annoying purchase of MTP) otherwise

Mellanox switches or cards?

In cards in Ethernet everything works fine, but in SX6036 switches I have tried Finisar FTL4C1QL2L LR4-Lite and FS.com modules and both only worked after I patched the vendor to "Mellanox"
Even worse ConnectX-5 and Infiniband, because the CX-5 seems to do a deeper inspection and still detects the modules as "FS" whereas the switch happily reports "Mellanox"

 

[crackelf]

Would you mind sharing this CLI command? I have also had a fair amount of issues with SX6036 + LR4 Transceivers in the last couple of days (result: only Mellanox branded / programmed LR4 optics seem to be accepted by the SX6036)

Having that said, LR4 optics + ConnectX-5 are also a bit problematic. In Infiniband mode, the card only links up to SDR (10Gbit)...


Mellanox switches or cards?

In cards in Ethernet everything works fine, but in SX6036 switches I have tried Finisar FTL4C1QL2L LR4-Lite and FS.com modules and both only worked after I patched the vendor to "Mellanox"
Even worse ConnectX-5 and Infiniband, because the CX-5 seems to do a deeper inspection and still detects the modules as "FS" whereas the switch happily reports "Mellanox"

This has me thinking I need specific modules too. My cards link out to the X-5 page even though they're allegedly 4's... Reassuring someone else is having issues with this too. Thanks for the report! I have some Mellanox DACs coming soon to test.

 

[crackelf]

Here's the NICs in question in case this helps anyone else
MCX4131A-GCAT C05
CX4131A

http://imgur.com/ILu9nwK

http://imgur.com/ekRSs3R

 

[crackelf]

Looping in this post from @i386

I was hoping to see something like hpe typcal product numbers eg "123456-B85". But the numbers look like "vanilla" mellanox nics which is confusing becuase the sellers says that the psid was changed (which usually is done to be able to flash stock mellanox firmware over oem firmware)

Another idea or possible cause: revision a1 + "made in Israel" could be an earlier version (or engineering sample). All the connectx-4 nics that I have are a2 or newer and the hpe branded models are all "made in China", the vanilla cx-4 are all "made in Israel"

Hopefully by the end of this week or middle of next I'll have answers for what exactly is happening here.

 

[crackelf]

Got some X3 VPI cards in

Configurations:                              Next Boot       New
         LINK_TYPE_P1                        N/A             ETH(2)       

Apply new Configuration? (y/n) [n] : y
Applying... Failed!
-E- Failed to set configuration: Failed to get default parameters: Operation not supported please specify all the parameters for VPI settings.

echo eth > /sys/devices/pcixxxx/xx/mlx4_port1 enables the eth, but I still can't run the param change

 

[i386]

Vpi?
I thought that's a LX nic (Ethernet only)

 

[crackelf]

Vpi?
I thought that's a LX nic (Ethernet only)

Just got some VPI cards in for testing. Edited above for clarity.

 

[prdtabim]

Just got some VPI cards in for testing. Edited above for clarity.

I verify and sysfs is read only in the mlx4_core module.
To configure the card you will pass module params or in some cases use mlxconfig .

 

[fohdeesha]

Got some X3 VPI cards in

Configurations:                              Next Boot       New
         LINK_TYPE_P1                        N/A             ETH(2)      

Apply new Configuration? (y/n) [n] : y
Applying... Failed!
-E- Failed to set configuration: Failed to get default parameters: Operation not supported please specify all the parameters for VPI settings.

echo eth > /sys/devices/pcixxxx/xx/mlx4_port1 enables the eth, but I still can't run the param change

mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

https://forums.servethehome.com/index.php?threads/flashing-stock-mellanox-firmware-to-oem-emc-connectx-3-ib-ethernet-dual-port-qsfp-adapter.20525/#post-198015

 

[crackelf]

I verify and sysfs is read only in the mlx4_core module.
To configure the card you will pass module params or in some cases use mlxconfig .

Yes in the end I went the module route for part of it, sysfs for the eth enablement, and the mellanox config tool for SRIOV. Strange combination.
/etc/modprobe.d/mlx4_core.conf looks like

options mlx4_core num_vfs=2 port_type_array=2 probe_vf=2

mstconfig -d set SRIOV_EN=1 NUM_OF_VFS=2
and

echo eth > /sys/devices/pcixxxx/xx/mlx4_port1

 

[crackelf]

mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

https://forums.servethehome.com/index.php?threads/flashing-stock-mellanox-firmware-to-oem-emc-connectx-3-ib-ethernet-dual-port-qsfp-adapter.20525/#post-198015

Still not able to get this to stick, but thanks for the link! Somehow I've gotten SRIOV working and ethernet enabled despite the firmware disagreeing. See quote. Will try flashing / updating firmware to see if that changes anything. See you in the other thread

I verify and sysfs is read only in the mlx4_core module.
To configure the card you will pass module params or in some cases use mlxconfig .

Yes in the end I went the module route for part of it, sysfs for the eth enablement, and the mellanox config tool for SRIOV. Strange combination.
/etc/modprobe.d/mlx4_core.conf looks like

options mlx4_core num_vfs=2 port_type_array=2 probe_vf=2

mstconfig -d set SRIOV_EN=1 NUM_OF_VFS=2
and

echo eth > /sys/devices/pcixxxx/xx/mlx4_port1

 

[crackelf]

(result: only Mellanox branded / programmed LR4 optics seem to be accepted by the SX6036)

You ended up being right! Mellanox transceivers are all I've been able to get working with these.

I'll make a support matrix next week after some testing and post it here. Thank you for the insight. I would never have guessed.