mellanox sx6036 and cx3 cards

thetoad

Active Member
Feb 10, 2021
236
97
28
I have an sx6036 that has been licensed for VPI and I have CX3 cards that I have flashed with the latest firmware and can put their 2 ports into ethernet mode (can see the devices come up in dmesg and ethtool enumerates them just fine).

however, none of the direct attach cables I have (including mellanox ones) seem to create a link between the card and the switch. (switch is in vpi mode, pondering changing it to ethernet only mode).

I've tried with multiple cables in both ports of the cx3 and multiple ports on the switch.

any idea if I'm doing something stupidly wrong or what i should look at?
 

Rand__

Well-Known Member
Mar 6, 2014
6,404
1,632
113
I have an sx6036 that has been licensed for VPI and I have CX3 cards that I have flashed with the latest firmware and can put their 2 ports into ethernet mode (can see the devices come up in dmesg and ethtool enumerates them just fine).
Have you activated the GW license?
I.e. are the ports of the 6036 in ETH mode?
1638470796022.png
 

thetoad

Active Member
Feb 10, 2021
236
97
28
first, I believe, its fully licensed

Feature EFM_SX
Description Generic SX license
Valid yes
Active yes
Eth L2 enabled true (ok)
Eth enabled true (ok)
Eth L3 enabled true (ok)
GW ports number 36 (ok)
Eth port SW speed limit 100Gb (ok)
IB port SW speed limit EDR (ok)
Max num ufm ports supported 36 (ok)
FCF enabled true (ok)


second, this is what ethool shows for the port


Code:
ethtool enp67s0
Settings for enp67s0:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseKX/Full
                            10000baseKX4/Full
                            10000baseKR/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
                            56000baseCR4/Full
                            56000baseSR4/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  1000baseKX/Full
                            10000baseKX4/Full
                            10000baseKR/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: off
    Port: FIBRE
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
    Link detected: no
does that look reasonable? should it be fibre? or would that be why my DACs dont work?

so, now I do see my transceiver for at least one cable in the switch, but you're right port is in ib mode, how do I switch it to ethernet mode?


Port Info


Port number : 5
Port type : IB




Identifier : QSFP+
Cable/ Module type : Passive copper, unequalized
Infiniband speeds : SDR , DDR , QDR , FDR
Vendor : Mellanox




Cable length : 1 m
Part number : 00W0049
Revision : A1
Serial number :
 

thetoad

Active Member
Feb 10, 2021
236
97
28
and after finding the protocol type submenu on ports and switch 5 to ethernet, it now is alive!

Code:
ethtool enp67s0d1
Settings for enp67s0d1:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseKX/Full
                            10000baseKX4/Full
                            10000baseKR/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
                            56000baseCR4/Full
                            56000baseSR4/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  1000baseKX/Full
                            10000baseKX4/Full
                            10000baseKR/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Link partner advertised link modes:  40000baseCR4/Full
    Link partner advertised pause frame use: No
    Link partner advertised auto-negotiation: Yes
    Link partner advertised FEC modes: Not reported
    Speed: 40000Mb/s
    Duplex: Full
    Auto-negotiation: on
    Port: Direct Attach Copper
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
    Link detected: yes
 

thetoad

Active Member
Feb 10, 2021
236
97
28
any long cisco cable also works

Identifier : QSFP+
Cable/ Module type : Active copper, near and far end
Ethernet speed and type : Not supported
Vendor : CISCO



Cable length : 10 m
Part number : 1111619001
Revision : 03
Serial number :
 

thetoad

Active Member
Feb 10, 2021
236
97
28
only issue, its very noisy for my apt. this is really something that needs a house lab more than simply a home lab. worse comes to worse I'll sell it. ironically, mellanox equipment resells for more in Israel than it does in the USA :)
 
  • Haha
Reactions: tinfoil3d

Rand__

Well-Known Member
Mar 6, 2014
6,404
1,632
113
Yeah its not low noise...

And I can imagine prices if I look at what these guys want for untested, might not be working, ex Engineering sample or too dented to sell stuff from the labs (2 grands+)
 

MichalPL

Member
Feb 10, 2019
170
14
18
Stupid question but no experience with Mellanox switches - is Ethernet license build into sx6036 by default? or if found something on ebay it might not have it?
 

thetoad

Active Member
Feb 10, 2021
236
97
28
no, their firmware package contains a keygen (literally) that one has to pass a secret to to generate valid keys. It just so happens that they have a license key validator also in their firmware that has the secret embedded in it. Also happens that they don't really do anything to obfuscate the usage of that the secret in the validator, so if one ltraces the validator while validating an invalid key, one will see it pop out clearly (I think it was something like strdup("SECRET")

the only slightly complicated part is that the SX6036 is a PPC build, while most of us can't easily run PPC binaries (though I actually did get it to run via QEMU's PPC emulator). It turns out though that the same exists for their newer x86 hardware and license key generation/validation hasn't changed. In practice, people here don't post the keys publicly to avoid accusations of enabling license violations, but if one is competent, the instructions I gave, should send you down the proper path.
 
  • Like
Reactions: tinfoil3d

Rand__

Well-Known Member
Mar 6, 2014
6,404
1,632
113
Just make sure you dont get a -5 model (6005,6015,6025), and none of the 5xxx or SB7xxx can do Ethernet either