mellanox sx6036 and cx3 cards

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

thetoad

Active Member
Feb 10, 2021
236
102
28
I have an sx6036 that has been licensed for VPI and I have CX3 cards that I have flashed with the latest firmware and can put their 2 ports into ethernet mode (can see the devices come up in dmesg and ethtool enumerates them just fine).

however, none of the direct attach cables I have (including mellanox ones) seem to create a link between the card and the switch. (switch is in vpi mode, pondering changing it to ethernet only mode).

I've tried with multiple cables in both ports of the cx3 and multiple ports on the switch.

any idea if I'm doing something stupidly wrong or what i should look at?
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
I have an sx6036 that has been licensed for VPI and I have CX3 cards that I have flashed with the latest firmware and can put their 2 ports into ethernet mode (can see the devices come up in dmesg and ethtool enumerates them just fine).
Have you activated the GW license?
I.e. are the ports of the 6036 in ETH mode?
1638470796022.png
 

thetoad

Active Member
Feb 10, 2021
236
102
28
first, I believe, its fully licensed

Feature EFM_SX
Description Generic SX license
Valid yes
Active yes
Eth L2 enabled true (ok)
Eth enabled true (ok)
Eth L3 enabled true (ok)
GW ports number 36 (ok)
Eth port SW speed limit 100Gb (ok)
IB port SW speed limit EDR (ok)
Max num ufm ports supported 36 (ok)
FCF enabled true (ok)


second, this is what ethool shows for the port


Code:
ethtool enp67s0
Settings for enp67s0:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseKX/Full
                            10000baseKX4/Full
                            10000baseKR/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
                            56000baseCR4/Full
                            56000baseSR4/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  1000baseKX/Full
                            10000baseKX4/Full
                            10000baseKR/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Auto-negotiation: off
    Port: FIBRE
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
    Link detected: no
does that look reasonable? should it be fibre? or would that be why my DACs dont work?

so, now I do see my transceiver for at least one cable in the switch, but you're right port is in ib mode, how do I switch it to ethernet mode?


Port Info


Port number : 5
Port type : IB




Identifier : QSFP+
Cable/ Module type : Passive copper, unequalized
Infiniband speeds : SDR , DDR , QDR , FDR
Vendor : Mellanox




Cable length : 1 m
Part number : 00W0049
Revision : A1
Serial number :
 

thetoad

Active Member
Feb 10, 2021
236
102
28
and after finding the protocol type submenu on ports and switch 5 to ethernet, it now is alive!

Code:
ethtool enp67s0d1
Settings for enp67s0d1:
    Supported ports: [ FIBRE ]
    Supported link modes:   1000baseKX/Full
                            10000baseKX4/Full
                            10000baseKR/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
                            56000baseCR4/Full
                            56000baseSR4/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  1000baseKX/Full
                            10000baseKX4/Full
                            10000baseKR/Full
                            40000baseCR4/Full
                            40000baseSR4/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Link partner advertised link modes:  40000baseCR4/Full
    Link partner advertised pause frame use: No
    Link partner advertised auto-negotiation: Yes
    Link partner advertised FEC modes: Not reported
    Speed: 40000Mb/s
    Duplex: Full
    Auto-negotiation: on
    Port: Direct Attach Copper
    PHYAD: 0
    Transceiver: internal
    Supports Wake-on: d
    Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
    Link detected: yes
 

thetoad

Active Member
Feb 10, 2021
236
102
28
any long cisco cable also works

Identifier : QSFP+
Cable/ Module type : Active copper, near and far end
Ethernet speed and type : Not supported
Vendor : CISCO



Cable length : 10 m
Part number : 1111619001
Revision : 03
Serial number :
 

thetoad

Active Member
Feb 10, 2021
236
102
28
only issue, its very noisy for my apt. this is really something that needs a house lab more than simply a home lab. worse comes to worse I'll sell it. ironically, mellanox equipment resells for more in Israel than it does in the USA :)
 
  • Haha
Reactions: tinfoil3d

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Yeah its not low noise...

And I can imagine prices if I look at what these guys want for untested, might not be working, ex Engineering sample or too dented to sell stuff from the labs (2 grands+)
 

MichalPL

Active Member
Feb 10, 2019
189
25
28
Stupid question but no experience with Mellanox switches - is Ethernet license build into sx6036 by default? or if found something on ebay it might not have it?
 

thetoad

Active Member
Feb 10, 2021
236
102
28
no, their firmware package contains a keygen (literally) that one has to pass a secret to to generate valid keys. It just so happens that they have a license key validator also in their firmware that has the secret embedded in it. Also happens that they don't really do anything to obfuscate the usage of that the secret in the validator, so if one ltraces the validator while validating an invalid key, one will see it pop out clearly (I think it was something like strdup("SECRET")

the only slightly complicated part is that the SX6036 is a PPC build, while most of us can't easily run PPC binaries (though I actually did get it to run via QEMU's PPC emulator). It turns out though that the same exists for their newer x86 hardware and license key generation/validation hasn't changed. In practice, people here don't post the keys publicly to avoid accusations of enabling license violations, but if one is competent, the instructions I gave, should send you down the proper path.
 
  • Like
Reactions: tinfoil3d

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Just make sure you dont get a -5 model (6005,6015,6025), and none of the 5xxx or SB7xxx can do Ethernet either
 

P0rt4lN3T

Member
Nov 10, 2023
46
0
6
Hi guys

sorry for posting in such an old thread.. ok so i have a SX6036 it has the ethernet mode gateway license..

i have entered the ssh and changed th profile to ethernet.. i have a 40G 0.5m dac cable.. and i have couple of milijet ONTI GBICs QSFP28 40G and 100G that i have tested in other hardware running fine in ethernet mode.. but on the switch state allways down.. attached below some pictures.. 1710605525710.png


1710605563864.png

1710605659180.png
1710605693848.png


I am kind of lost now because none is UP and the leds on the switch ports not comming up neither.. do i need to change any other config? or do i need compatible 40/56GB GBICS ?

EDIT: BTW all the ports including the DAC cable stating high power transceiver is not supported.
 

P0rt4lN3T

Member
Nov 10, 2023
46
0
6
high power transceivers (in default configuration) only works in 4 ports (https://forums.servethehome.com/index.php?threads/mellanox-sx6036-compatibility.39247/post-368454)
Hiya thanks first of all after reading some posts i found this cmd fae cable-stamping-unlock 40g_lr4 after running the cmd via ssh putty on the switch, the DAC cable 40G started working.. on port 1,3 not sure if it will work on all the ports.. will make some testing on the Gbics 40Gbp Single mode double LC connector 10KM distance..