Mellanox Switches - Tips & Tricks

[mpchris]

Each image file includes an MFG system (you don’t have to use the MFG version that matches the image).

After booting to MFG, run the included manufacture.sh script, this does not require installing ONIE. For switches that use the traditional boot method, please use this installation approach.

Ohh interesting, how do I boot into MFG tho? I see that the img contains the mfg-initrd and mfg-kernel but my brain is too smooth right now to figure out what to do with that :/
When I boot to single user mode I can find the manufacture.sh in /sbin, but I don't think that is the right way to run this script?

 

[Tim111]

Each image file includes an MFG system (you don’t have to use the MFG version that matches the image).

After booting to MFG, run the included manufacture.sh script, this does not require installing ONIE. For switches that use the traditional boot method, please use this installation approach.

Can this be done from single user mode?
He says he can't get to regular shell.

 

[Tim111]

Ohh interesting, how do I boot into MFG tho? I see that the img contains the mfg-initrd and mfg-kernel but my brain is too smooth right now to figure out what to do with that :/
When I boot to single user mode I can find the manufacture.sh in /sbin, but I don't think that is the right way to run this script?

sh ./sbin/manufacture.sh

 

[alexv92]

Hi everyone,

I’m trying to get RoCEv2 over NVMe-oF working through an SN2100 switch, and I’ve hit a wall.
Direct connection (no switch):

rping and nvme discover work perfectly on all hosts.


Through SN2100:

rping -c -a <IP> fails with RDMA_CM_EVENT_UNREACHABLE, error -110
nvme discover times out
BUT: iperf3 works at ~98 Gbps → basic L2/L3 forwarding is fine


Hosts:

Mellanox ConnectX-6 Dx EN (MCX623106AN-CDA_Ax)
Ubuntu 22.04 / 24.04 / 25.04
All ports configured for RoCEv2:
ROCE_CONTROL = ROCE_ENABLE(2)
QOS_TRUST_STATE = TRUST_DSCP(2)
ROCE_RTT_RESP_DSCP = 26, CNP_DSCP = 48
ROCE_CC_PRIO_MASK = 32 (for DSCP 26 → prio 5)
LLDP_NB_* = ALL, DCBX = True
MTU = 9216
IPv4-mapped GID ( ::ffff:10.20.x.x) present



Tested firmware versions (none worked):

3.10.2102
3.10.4006
3.10.4400
3.10.4606
3.10.2102
3.7.1134
3.9.3302


All of them result in the same UNREACHABLE error.

Has anyone successfully run RoCEv2 through an SN2100 switch? If yes, what ONYX version did you use?
Is there a known workaround for this issue in dev builds?
Would Cumulus Linux 5.5+ be a reliable alternative for RoCE/NVMe-oF on SN2100?


Any advice or similar experiences would be greatly appreciated!

 

[mpchris]

sh ./sbin/manufacture.sh

No I meant if single user mode is the right environment to run this script, I generally know how to run scripts
It probably isn't, also the script needs some arguments.

 

[Tim111]

No I meant if single user mode is the right environment to run this script, I generally know how to run scripts
It probably isn't, also the script needs some arguments.

I never ran it nasbdh9 might know what you need but I would think yes to single user mode.

 

[edo1]

FYI, the update sequence and show asic-version values when updating an old SN2410 to current version. You cannot directly upgrade a switch with 3.7 on it to 3.10, as 3.10 will fail to recognise the ASIC

which upgrade path should I use to upgrade SN2010 from 3.6.8008?
where can I find intermediate versions?

using bruteforce and plan on https://support.hpe.com/connect/s/s...tes&softwareId=MTX_0eac2e961a714e23a6aafe2f55
has made such upgrade route:

https://content.mellanox.com/Software/image-X86_64-3.7.1134.img

https://content.mellanox.com/Software/image-X86_64-3.8.2204.img

https://content.mellanox.com/Software/image-X86_64-3.9.1020.img

https://content.mellanox.com/Software/image-X86_64-3.9.3302.img

https://content.mellanox.com/Software/image-X86_64-3.10.2102.img

3.10.4504 https://forums.servethehome.com/index.php?threads/mellanox-switches-tips-tricks.39394/post-487179
3.10.4606 https://forums.servethehome.com/index.php?threads/mellanox-switches-tips-tricks.39394/post-469194

 

[luckylinux]

What is the Idle Power consumption for a Mellanox SX6720 (without [almost] any DAC/Transceiver attached) ?

Is there some similar License that can activate Ethernet, as is the Case for the SX6036 ?

 

[NablaSquaredG]

Is there some similar License that can activate Ethernet, as is the Case for the SX6036 ?

You can in fact use the same license as for SX6036

What is the Idle Power consumption for a Mellanox SX6720 (without [almost] any DAC/Transceiver attached) ?

Roughly 10-15W more than a SX6036 if I remember correctly

 

[untergeek]

Is there some similar License that can activate Ethernet, as is the Case for the SX6036 ?

The SX6710 and ostensibly the SX6720 should have the Ethernet license by default. You might need a license to access the `_shell`, but not Ethernet. My SX6710G had it by default.

 

[Tim111]

I bought 2 sx6720 no license installed. To me way louder then sx6036 but control plane is faster. The sx6710g is gateway so has license installed same as sx6036g.

 

[luckylinux]

You can in fact use the same license as for SX6036

Are you sure ?

Roughly 10-15W more than a SX6036 if I remember correctly

Uhm OK, so maybe better to get some more SX6036 instead .

 

[Tim111]

Are you sure ?


Uhm OK, so maybe better to get some more SX6036 instead .

Yes same license.

 

[lllllmmm]

It appears Onyx has a new version: v3.10.4706, but I haven't found any information about it on Google other than the NVIDIA documentation. Could someone share this firmware?

 

[BackupProphet]

dont run a firmware no one else is running, we cant help you when you run something else than us

 

[tros10]

Hi
Please, somebody DM me with X86_64-3.10.4206-installer.bin link.
Have to do few S2700 with SSD replacement.

 

[nasbdh9]

If you are replacing your SSD, please do not install version 3.10 directly, as you will most likely encounter a problem where the firmware cannot be updated, causing the system to fail to load.

 

[tros10]

Im replacing old Innodisk SSDs, system are already updated to 3.10, so, no issues expected here. Still looking for X86_64-3.10.4206-installer.bin

 

[Scott Laird]

Hey, I have a SN3800 that I'd like to use with a stock Linux distribution (using it for VyOS dev), but it's currently running too old of an OS (3.8.2204) for the Linux `mlxsw` driver to be able to upgrade it:

[   30.449077] mlxsw_spectrum2 0000:06:00.0: The firmware version 29.2000.2698 is incompatible with the driver (required >= 29.2010.1006)

I jumped through the same hoops with a SN2010, and upgrading it to Onyx 3.9.x was good enough to get a firmware version that Linux could cope with, but every discussion that I see here is for Spectrum(1) Onyx images, and the SN3800 is a Spectrum2. Do they use the same images? If not, does anyone have a link to a 3.9.x Spectrum2 image that I can use for upgrading?

 

[nasbdh9]

Hey, I have a SN3800 that I'd like to use with a stock Linux distribution (using it for VyOS dev), but it's currently running too old of an OS (3.8.2204) for the Linux `mlxsw` driver to be able to upgrade it:

[   30.449077] mlxsw_spectrum2 0000:06:00.0: The firmware version 29.2000.2698 is incompatible with the driver (required >= 29.2010.1006)

I jumped through the same hoops with a SN2010, and upgrading it to Onyx 3.9.x was good enough to get a firmware version that Linux could cope with, but every discussion that I see here is for Spectrum(1) Onyx images, and the SN3800 is a Spectrum2. Do they use the same images? If not, does anyone have a link to a 3.9.x Spectrum2 image that I can use for upgrading?

mellanox - kernel/git/firmware/linux-firmware.git - Repository of firmware blobs for use with the Linux kernel

git.kernel.org

Updating Firmware

Contribute to Mellanox/mlxsw development by creating an account on GitHub.

github.com