How to use the mlx5_core driver with Mellanox ConnectX-4 Lx in Debian?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
C

crackelf

Member
Apr 11, 2021
74
6
8
i386 said:
Vpi?
I thought that's a LX nic (Ethernet only)
Click to expand...
ConnectX-4 Lx do not work with KAIAM modules:
Code: 
01:00.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [15b3:1015]
    Kernel driver in use: mlx5_core
    Kernel modules: mlx5_core
Code: 
show media ethernet 1/3/1
Port   1/3/1: Type  : 40GE-LR4   2km (QSFP+ LC)
         Vendor: KAIAM CORP           Version: 1A
         Part# : XQX2502              Serial#: KD60315209
Code: 
Settings for enp1s0np0:
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    Port: FIBRE
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000004 (4)
                               link
    Link detected: no
Proof that Mellanox programmed transceivers work:
Code: 
show media ethernet 1/4/1
Port   1/4/1: Type  : 40GBASE-Passive Copper  2m (QSFP+)
         Vendor: Mellanox             Version: A3
         Part# : MC2207130-002        Serial#: MT1433VS02614
Code: 
Settings for enp1s0np0:
    Speed: 40000Mb/s
    Duplex: Full
    Auto-negotiation: on
    Port: Direct Attach Copper
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000004 (4)
                               link
    Link detected: yes
Proof that KAIAM transceivers work properly with ConnectX-3 cards:
Code: 
02:00.0 Network controller [0280]: Mellanox Technologies MT27500 Family [ConnectX-3] [15b3:1003]
    Kernel driver in use: mlx4_core
    Kernel modules: mlx4_core
Code: 
show media ethernet 1/3/1
Port   1/3/1: Type  : 40GE-LR4   2km (QSFP+ LC)
         Vendor: KAIAM CORP           Version: 1A
         Part# : XQX2502              Serial#: KD60315209
Code: 
Settings for enp2s0:
    Speed: 40000Mb/s
    Duplex: Full
    Auto-negotiation: off
    Port: FIBRE
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
    Link detected: yes
 
P

prdtabim

Active Member
Jan 29, 2022
184
72
28
crackelf said:
ConnectX-4 Lx do not work with KAIAM modules:
Code: 
01:00.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [15b3:1015]
    Kernel driver in use: mlx5_core
    Kernel modules: mlx5_core
Code: 
show media ethernet 1/3/1
Port   1/3/1: Type  : 40GE-LR4   2km (QSFP+ LC)
         Vendor: KAIAM CORP           Version: 1A
         Part# : XQX2502              Serial#: KD60315209
Code: 
Settings for enp1s0np0:
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: on
    Port: FIBRE
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000004 (4)
                               link
    Link detected: no
Proof that Mellanox programmed transceivers work:
Code: 
show media ethernet 1/4/1
Port   1/4/1: Type  : 40GBASE-Passive Copper  2m (QSFP+)
         Vendor: Mellanox             Version: A3
         Part# : MC2207130-002        Serial#: MT1433VS02614
Code: 
Settings for enp1s0np0:
    Speed: 40000Mb/s
    Duplex: Full
    Auto-negotiation: on
    Port: Direct Attach Copper
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000004 (4)
                               link
    Link detected: yes
Proof that KAIAM transceivers work properly with ConnectX-3 cards:
Code: 
02:00.0 Network controller [0280]: Mellanox Technologies MT27500 Family [ConnectX-3] [15b3:1003]
    Kernel driver in use: mlx4_core
    Kernel modules: mlx4_core
Code: 
show media ethernet 1/3/1
Port   1/3/1: Type  : 40GE-LR4   2km (QSFP+ LC)
         Vendor: KAIAM CORP           Version: 1A
         Part# : XQX2502              Serial#: KD60315209
Code: 
Settings for enp2s0:
    Speed: 40000Mb/s
    Duplex: Full
    Auto-negotiation: off
    Port: FIBRE
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
    Link detected: yes
Click to expand...
Just to be clear. Are the tests above between a pcie card ( connectx-4 / connectx-3 ) and a switch port with 2 KAIAM optical modules ? is the conclusion that KAIAM module isn't compatible with Connectx-4 ?
 
N

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,540
996
113
crackelf said:
You ended up being right! Mellanox transceivers are all I've been able to get working with these.

I'll make a support matrix next week after some testing and post it here. Thank you for the insight. I would never have guessed.
Click to expand...
You can remove the lock with a magic CLI command

Re 40GBase Transceivers in Mellanox:
I will receive some Kaiam modules (the ones from the Brocade Thread) soon.

I can at least say that 40GBase-LR Finisar modules work in Ethernet Mode (40G/s link) in ConnectX-5, BUT in Infiniband mode only Mellanox coded transceivers are accepted, otherwise link is 10G (SDR)

I am currently waiting for fs.com to get back to me regarding my request of custom programmings for fs.com transceivers - Can't wait to see whether they manage to create a programming that is compatible with ConnectX-4 / ConnectX-5
 
C

crackelf

Member
Apr 11, 2021
74
6
8
prdtabim said:
Just to be clear. Are the tests above between a pcie card ( connectx-4 / connectx-3 ) and a switch port with 2 KAIAM optical modules ? is the conclusion that KAIAM module isn't compatible with Connectx-4 ?
Click to expand...
Yes, precisely. This is between a PCIE card and a Brocade ICX 7450 switch port with 2 KAIAM optical modules.

Even directly connecting does not work between two ConnectX-4 Lx cards using the KAIAM modules.
 
C

crackelf

Member
Apr 11, 2021
74
6
8
NablaSquaredG said:
You can remove the lock with a magic CLI command
Click to expand...
Is the CLI for switches or for PCIE cards? I'm talking about PCIE cards.
NablaSquaredG said:
Re 40GBase Transceivers in Mellanox:
I can at least say that 40GBase-LR Finisar modules work in Ethernet Mode (40G/s link) in ConnectX-5, BUT in Infiniband mode only Mellanox coded transceivers are accepted, otherwise link is 10G (SDR)
Click to expand...
Good to know. I haven't gotten mine to connect via Ethernet Mode with anything other than Mellanox, but maybe the Lx variant of my card is slightly different than your ConnectX-5. The drivers link out the same, but I suspect that may be wrong.
NablaSquaredG said:
I am currently waiting for fs.com to get back to me regarding my request of custom programmings for fs.com transceivers - Can't wait to see whether they manage to create a programming that is compatible with ConnectX-4 / ConnectX-5
Click to expand...
They're pretty impressive. I wouldn't be surprised if they can fulfill that request. Good luck and please report back with your findings!
 
C

crackelf

Member
Apr 11, 2021
74
6
8
prdtabim said:
Just to be clear. Are the tests above between a pcie card ( connectx-4 / connectx-3 ) and a switch port with 2 KAIAM optical modules ? is the conclusion that KAIAM module isn't compatible with Connectx-4 ?
Click to expand...
Final results that I'm willing to try out: Brocade modules do not work with this model of ConnectX-4 either. The only thing I can get working is Mellanox programmed transceivers.
crackelf said:
MCX4131A-GCAT C05
CX4131A
http://imgur.com/ILu9nwK
http://imgur.com/ekRSs3R
Click to expand...
Code: 
Port   1/4/1:
         Vendor: RUCKUS               Version: A 
         Part# : 57-1000267-01        Serial#: HTA2150600000AY
Plugged into Mellanox ConnectX-4 Lx:
Code: 
        Supported ports: [  ]
        Supported link modes:   1000baseKX/Full
                                10000baseKR/Full
                                40000baseKR4/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                40000baseLR4/Full
                                25000baseCR/Full
                                25000baseKR/Full
                                25000baseSR/Full
                                50000baseCR2/Full
                                50000baseKR2/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: None        RS      BASER
        Advertised link modes:  1000baseKX/Full
                                10000baseKR/Full
                                40000baseKR4/Full
                                40000baseCR4/Full
                                40000baseSR4/Full
                                40000baseLR4/Full
                                25000baseCR/Full
                                25000baseKR/Full
                                25000baseSR/Full
                                50000baseCR2/Full
                                50000baseKR2/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: None       RS      BASER
        Speed: Unknown!
        Duplex: Unknown! (255)
        Auto-negotiation: on
        Port: Other
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000004 (4)
                               link
        Link detected: no (No cable)
Plugged into ConnectX-3
Code: 
    Supported ports: [ FIBRE ]
    Supported link modes:   10000baseKX4/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
                            56000baseCR4/Full
                            56000baseSR4/Full
                            1000baseX/Full
                            10000baseCR/Full
                            10000baseSR/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10000baseKX4/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
                            1000baseX/Full
                            10000baseCR/Full
                            10000baseSR/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: 40000Mb/s
    Duplex: Full
    Auto-negotiation: off
    Port: FIBRE
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
    Link detected: yes
A huge thank you to everyone who was helpful along the way. If it weren't for this post I wouldn't have tried the Mellanox programmed transceivers:
NablaSquaredG said:
(result: only Mellanox branded / programmed LR4 optics seem to be accepted by the SX6036)
...I have tried Finisar FTL4C1QL2L LR4-Lite and FS.com modules and both only worked after I patched the vendor to "Mellanox"
Even worse ConnectX-5 and Infiniband, because the CX-5 seems to do a deeper inspection and still detects the modules as "FS" whereas the switch happily reports "Mellanox"
Click to expand...
Final support matrix for these cards with Brocade ICX 7450-48p = Mellanox transceiver modules only.
 
You must log in or register to reply here.
Top Bottom