Mellanox Switches - Tips & Tricks

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,836
1,215
113
Does it have some onboard flash separate than the SSD?
Yes.

The switch has an EEPROM (which stores ONIE info) and some model specific Onyx info (in the ONIE vendor extensions).

Onyx will see this and use the appropriate vendor customization file (and apply port licensing if applicable for this model)
 

soupyfrood

New Member
Dec 12, 2019
22
3
3
Would anyone please be able to advise how loud a SN2410 is?
Surprisingly, it's not very loud. I just got one yesterday. Measuring with an old Radio Shack SPL meter at 3 feet gave me a peak of 77dB during bootup. It dropped to 63dB after booting, and has how settled in at 55 dB. It's definitely making sound, but isn't terribly objectionable. Power draw is 33 watts. Both PSUs are connected, but it's only pulling power from PSU 1. According to fae show environmental the 8 chassis fans are spinning between 5400 and 6400 RPM.
 
  • Like
Reactions: DRAGONKZ

The_Never_Hawk

New Member
Mar 22, 2022
4
0
1
I am having difficulty getting some 40g modules to work on the SN2100, this module is working in SX6036 (despite showing up as unsupported) and a Mikrotik switch, but in the SN2100 it can only do 1g. I have tried the fae cable stamping unlock command. Mellanox official aoc works in 40g but not 56g.

Any ideas?
 

klui

༺༻
Feb 3, 2019
991
581
93
Either the SN2100 has similar limitations like the SX6036 that prevents high power modules from working on certain ports, or the same reason why CX4s and newer are more choosy about what vendors of optics they accept.
 

i386

Well-Known Member
Mar 18, 2016
4,625
1,757
113
36
Germany
CX4s and newer are more choosy about what vendors of optics they accept
They are more picky? I have brocade, broadcom, mellanox, fs.com, cisco, finisar branded 10 & 40 GBE transceivers (all short reach sfp or qsfp) and so far they all work between my cx-4 and arista, ruckus or mellanox switches
 

klui

༺༻
Feb 3, 2019
991
581
93
@NablaSquaredG mentioned it at https://forums.servethehome.com/index.php?threads/5-intel-100g-cwdm4-qsfp28.39518/post-373848 as well as in several other places. I tried and posted confirmation but can't find my post, which was a reply to another thread elsewhere on this site.

I recall non-Mellanox optics won't link at 40G in a CX4/CX5 with standard Mellanox firmware. Intel 100G optics worked fine at 100G but not 40G. I can't recall what I tested in addition to Intel, but probably Brocade and Arista.

What part #s are your Brocade? I only have Arista, Brocade, Intel, Juniper, and Mellanox optics.
 

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,836
1,215
113
I recall non-Mellanox optics won't link at 40G in a CX4/CX5 with standard Mellanox firmware.
I think it's only 100G running at 40G that require the Mellanox OUI to be set in the transceiver EEPROM. 40G should run at 40G no problem, even if the Mellanox OUI is not set.
 

The_Never_Hawk

New Member
Mar 22, 2022
4
0
1
Either the SN2100 has similar limitations like the SX6036 that prevents high power modules from working on certain ports, or the same reason why CX4s and newer are more choosy about what vendors of optics they accept.
If the first explanation is true, then why does it work on the SX6036? On the SX6036 it shows module not supported, but worked anyway.

If the latter explanation is true, then why did this work on my CX5 and CX6Dx?

Also, on Mikrotik this module reports as 40g base sr4
 

klui

༺༻
Feb 3, 2019
991
581
93
The answer is obvious. Different platform/firmware. Same thing with my Aruba S2500: unsupported optics but they function. Why? I don't know but it's just a function of the firmware by the individual who committed it at that time.

But since I don't have source code for Mellanox switch or firmware, I'm just offering speculation.
 

Bilbo

New Member
Feb 19, 2016
24
10
3
Hello All,

I am sharing the file below to anyone looking to get started with Mellanox switches. It includes the following payload.

- A folder with everything you need to reinstall ONIE on an x86 (SN series switch)
- Instructions on how to use rufus (for the onie install) and the key commands you need to fully install a fresh OS
- A .bat tool kit that I built which uses user prompts to generate clipboard text to quick generate SSH commands to setup your switch

NOTES

- These files will only work on SN series switches. They WILL NOT work on PPC switch variants and I do not recommend trying this on infinaband / SX switches. Some might argue with me, but I wouldn't even mess with the SX switches at all since they are a little more finicky and harder to find compatible resources for.
- Read the instructions for how to use the install files first before you begin anything
- The MLNX_CYOA.bat file is all you need to run to get access to the config generators. I think I have most of the bugs out of it, but it is obviously use at your own risk. I would past the clipboard into a txt editor first and review the config before entering it into the switch. This tool only spits out syntax for ONYX OS, not cumulus.
- DM me if you have questions

 
Last edited:

up3up4

Member
Jun 10, 2018
90
33
18
First you have to copy it to your own Dropbox account then you will have to download it from your account.
 

svvolf

New Member
Mar 30, 2017
16
7
3
41
This is the latest version, previously the download link on the website was not open for anonymous guests to download, maybe I sent an email requesting to open the download before and they received it and allowed it, so now you can download it for free, plus I've provided my share link for more people to download it.

onyx-X86_64-3.10.4606.img
md5:d54d379a11621ea2b22a23c08917397f
 
  • Like
Reactions: SGS and klui

lllllmmm

New Member
Jul 14, 2023
15
3
3
This is the latest version, previously the download link on the website was not open for anonymous guests to download, maybe I sent an email requesting to open the download before and they received it and allowed it, so now you can download it for free, plus I've provided my share link for more people to download it.

onyx-X86_64-3.10.4606.img
md5:d54d379a11621ea2b22a23c08917397f
The release notes for onyx-3.10.4606 are also available on the HPE website. It looks like the LTS support for onyx has been extended to 2029

 
  • Like
Reactions: Freebsd1976

The_Never_Hawk

New Member
Mar 22, 2022
4
0
1
This is the latest version, previously the download link on the website was not open for anonymous guests to download, maybe I sent an email requesting to open the download before and they received it and allowed it, so now you can download it for free, plus I've provided my share link for more people to download it.

onyx-X86_64-3.10.4606.img
md5:d54d379a11621ea2b22a23c08917397f
I thought the Spectrum x86 switches only supported 3.10.4XXX, as per the first post of the thread first post of the thread


Spectrum x86 (SN2100, SN2700, SN2010, etc)3.10.4XXX (3.10.4404 as of 24th January 2024)
(DO NOT upgrade to 3.10.5000, 3.10.6004, 3.11.XXXX etc.)
 

Bilbo

New Member
Feb 19, 2016
24
10
3
According to the NVIDIA Support portal, the highest available target version is 3.10.4606

1748533991080.png