Mellanox Switches - Tips & Tricks

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

BoGs

Active Member
Feb 18, 2019
159
38
28
- Any of you know if a SX6036 railkit would work for SN2700? or 2410? Arista rails kit works for all models
- What would be the minimum fan speed SN2700 would be able to go?
- Reading the manual seems that the port split on top port disables the bottom port? Is that correct? I think saw photos in this thread to the opposite.
 

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,805
1,196
113
- Any of you know if a SX6036 railkit would work for SN2700? or 2410? Arista rails kit works for all models
Definitely not on the SN2700, because it has little knobs that stick out... Get the proper railkit.

- Reading the manual seems that the port split on top port disables the bottom port? Is that correct? I think saw photos in this thread to the opposite.
Depends on the split mode. 4x25G split mode will disable the bottom port, 2x50G will not.

- What would be the minimum fan speed SN2700 would be able to go?

5400 / 6300 RPM (those fans are dual counter rotating) at 30% PWM. They are comparably quiet at 5400 RPM, quieter than an Ruckus ICX7650-48ZP for example.
 
  • Like
Reactions: BoGs

adrisr

New Member
Feb 27, 2025
2
0
1
Hello, I'm new here, and I just want to thank everybody who have contributed to this thread, and to share my story with a pair of SB7700, just in case it is useful for anybody else:

I have a pair of SB7700 that have been powered on since one fellow of mine installed them almost 8 years ago. Recently, I had the need to move them from the rack where they were placed, so I had to turn them off. Of course, they didn't restart when I pluged them again. One of them showed in the console a message about not having any boot partition. The other one showed nothing at all.
After discovering this thread, I opened both of them. The one with no console output has a blown out capacitor, so that's the end for that one. In the other one, I was able to enter the bios, and after rebooting with the ONIE 2020.11-5.3.0005-115200 in a usb, I was able to start tinkering with it. I checked both of the SSDs, but they both were dead. I bought a Kinkston SSD, and I started the process that's described in this thread (for the SB7700, it seems that the right combination is CSM enabled + Non-UEFI ONIE, btw). I must mention that I was receiving some warnings related to the system not being capable of reading the serial number from TLV (or something like that), but the installation carried on. Searching about it I discovered that it was related to the EEPROM. I checked the content of the EEPROM with onie-syseeprom and it only had the verification checksum, all the other values were empty. But I carried on...
Even though in the first post it's said that the last version for the SB7700 is the 3.9.3124, I saw in the nvidia support page that they list the 3.9.3302 as the last supported version for the SB7700, so I tried with the installer located in the mega folder, the X86_64-3.9.3202-installer.bin. Unfortunately, after filling all the new partitions, near the end, the installation process failed with an error related to the mlxi2c command (MLXI2C_AUTO_DETECT_FAILED). However, I could boot MLNX-OS after the installation, but I had to wait for around 15 minutes from login in until I got to the prompt, and the output of 'show asic-version' showed that no managed switches were detected. I flashed many versions of mlnx-os via the update procedure, but none of them worked.
Then I started to suspect that the eeprom was more important than what I thought... I started to fill some fields with the information I could retrieve from the smbios, with dmidecode -t1. But the installation from onie to X86_64-3.9.3202-installer.bin always failed in the same place, with the same error (mlxi2c related). After more searches, I tried to fill some of the eeprom values that I had no idea how to fill... I located the onie-syseeprom dump from someone with a SN2100, and I noticed that there is a field Vendor Extension (0xfd) fields with hex numbers. I naively copied these fields, and after that the installation with the X86_64-3.9.3202-installer.bin worked... but it got all the values from a SN2100, so after restarting the system, mlnx-os still doesn't detect the asic. But I think that those values are the key to succesfully recover the switch.
Could anybody provide the onie-syseeprom dump from a working sb7700? Or maybe there is a way to get those values from the smbios and encode them in the right way, I don't know... Either way, any help would be greatly appreciated.
 

dbTH

Member
Apr 9, 2017
171
64
28
Onyx NOS will eventually reach end-of-life. I'm thinking of converting the SN2100M/SN2010M switch currently running Onyx to Cumulux Linux. Has any homelab user here already done this and know of a low-cost way to do it, especially on acquiring the Cumulux Linux license?
 
Last edited:

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,805
1,196
113
especially on acquiring the Cumulux Linux license?
Cumulus doesn’t need a license anymore since quite some time.

You can find installers publicly accessible in the internet if you search around a bit
 
  • Like
Reactions: dbTH

cy384

New Member
Aug 19, 2022
23
22
3
cy384.com
is it confirmed anywhere which, if any, revisions of the SN2100 or SN2010 have fixes for the Atom C2000 CPU bug? Or if there's a viable hackjob fix one can do? (I'm mostly interested in the SN2010)

I see revisions labelled like A1, A8, BF, BA, B2, BD, etc. on the pullout tab on ebay.
 

i386

Well-Known Member
Mar 18, 2016
4,588
1,737
113
35
Germany
Revisions I don't know, but I assume that everything manufactured after 2018 hast the new cpu revisions or the lpc fixes applied.
 

dbTH

Member
Apr 9, 2017
171
64
28
Well that's not exactly helpful for @cy384 I guess...
If any forum member has access to a NVIDIA Enterprise Support account with valid SN2xxx support, it would be best to open a support ticket with NVIDIA to confirm which revisions of the SN2010 and SN2100 are affected by the Atom C2000 CPU bug. Otherwise, we're just guessing.
 

cy384

New Member
Aug 19, 2022
23
22
3
cy384.com
If any forum member has access to a NVIDIA Enterprise Support account with valid SN2xxx support, it would be best to open a support ticket with NVIDIA to confirm which revisions of the SN2010 and SN2100 are affected by the Atom C2000 CPU bug. Otherwise, we're just guessing.
I wonder if we can crowdsource this. If anyone with 2100/2010s can share their labelled revision, plus check the CPU stepping ( lscpu or looking at /proc/cpuinfo ), we might be able to narrow it down.
 

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,805
1,196
113
I wonder if we can crowdsource this. If anyone with 2100/2010s can share their labelled revision, plus check the CPU stepping ( lscpu or looking at /proc/cpuinfo ), we might be able to narrow it down.
SN2010, 2x Rev A4, 1x Rev A8: All have C0 Atom and Spectrum A1
 
  • Like
Reactions: cy384

Freebsd1976

Active Member
Feb 23, 2018
410
75
28
I wonder if we can crowdsource this. If anyone with 2100/2010s can share their labelled revision, plus check the CPU stepping ( lscpu or looking at /proc/cpuinfo ), we might be able to narrow it down.
asked 4 year ago and never got response from them.
btw:
once have touched a new SN2010B Rev A5 which SN is MT165........(produced in 2016 ) , it is B0 step
 
Last edited:
  • Like
Reactions: cy384

Mario Gomes

New Member
Jan 22, 2020
5
0
1
Portugal
Cumulus doesn’t need a license anymore since quite some time.

You can find installers publicly accessible in the internet if you search around a bit
I've acquired 2 SN2700 for my home lab, and would like to give cumulus a try, I've searched quite extensively the net and couldn't find any installers, but maybe I'm not asking the right questions, would you be able to give me some pointers on where to look?
 

Mario Gomes

New Member
Jan 22, 2020
5
0
1
Portugal
I've acquired 2 SN2700 for my home lab, and would like to give cumulus a try, I've searched quite extensively the net and couldn't find any installers, but maybe I'm not asking the right questions, would you be able to give me some pointers on where to look?
Found version cumulus-linux-5.11.0-mlx-amd64.bin online, but it doesn't have checksum and I couldn't find the official checksum for this file to verify it's authenticity, does anyone have this information or has access to it?
 

BoGs

Active Member
Feb 18, 2019
159
38
28
My god I finally got my SN2700 setup swapped in a new SSD and finished upgrading the switch.

The pain associated with the usb keys is crazy, for some reason new rufus version uses large fat32 for usb 3 keys - which did not work for booting on it for Onie Recovery, I had to find a usb2 key which then worked fine and took a try or two until bios in 9600 switch to 115200 in mac os screen to get it to work before the Grub timeout happened and booted in recovery.

Also for anyone else do not use basic boot no UEFI boot, usb2 key made with rufus. I tried doing dd in mac and did not write the image, had lots of issues with that. Might have been my usb3 key but after using usb2 all is good now.