Similar to the glorious Brocade ICX Series (cheap & powerful 10gbE/40gbE switching) Thread, I'd like to use this thread to compile some knowledge about Mellanox switches that is spread over many places in the forum, which can be hard to find for newbies.
I will expand this thread / post over time, so if you have some information that you'd like to have added, just write it in a comment.
(MAXIMUM) SUPPORTED FIRMWARE VERSIONS:
Upgrading beyond these versions or using other versions may brick your switch and require a recovery procedure!
As I don't have a lot of time right now (but want to get this done before I forget it), I'll just start with some basics
Overview of current Mellanox switch series that may be interesting for homelabs:
- Mellanox IS5000 Series: Old, Infiniband only switches. They should generally be avoided:
- Mellanox SX Series: 40G / 56G switches with Switch-X / SwitchX-2 chip. They come in different flavours (unmanaged / managed, full width, half width, etc... TODO: Full Model Listing). They are very energy efficient (TODO: Exact numbers). Most of them are PowerPC, but some are x86
Managed SX Series switches can do VPI - That means you can have some ports do Infiniband, some Ethernet at the same time and even use integrated IPoIB gateway functionality.
Highest supported firmware version (for managed switches): 3.6.8012
DO NOT try to update to a version beyond that on x86 switches - It may brick your switch by automatically doing a bios update that prevents the switch ASIC from being detected!
- Mellanox SB77XX/78XX series: 100G Infiniband switches with Switch-IB / Switch-IB 2 chip (no VPI like SX Series). x86 control plane, highest supported version: 3.9.3124
Very power efficient! SB7700 needs only 53W in IDLE (one PSU)
- Mellanox SN2XXX series: 100G Ethernet switches with Spectrum chip (no VPI like SX series). x86 control plane, highest supported version: ?? Apparently, it's 3.10.4100 (LTS)
Very power efficient. SN2700 needs only 51W in IDLE (one PSU), SN2100 about 35W iirc (after the fans have spun down)
They come in different flavours, but generally you have:
- SN2700: 19" 32x100G, Celeron 1047U, mSATA SSD (TODO: Expand)
- SN2100: 9.5" (half-width), Atom C2XXX (possibly affected by AVR54 bug?), M.2 SATA SSD
... and more (TODO: Full model listing)
Tips & Tricks for working with those switches:
- SN2XXX / SB7XXX series: Replace your SSDs and make backups! The original Innodisk 3ME SSDs are prone to failure (one died while I was taking an image).
You can use whatever mSATA (apparently all except SN2100) or M.2 SATA (apparently only SN2100) you want. My go-to model are Transcend 452T2 (e.g. TS128GMSA452T2)
- If you get an SN2XXX with ONIE, Cumulus or no OS, you can easily flash it to ONYX / MLNX-OS by taking a good MLNX-OS / ONYX image from another switch and patching the embedded database to the correct model number, number of ports, MAC Addresses, etc... (TODO: Guide)
TODO: Expand
I will expand this thread / post over time, so if you have some information that you'd like to have added, just write it in a comment.
(MAXIMUM) SUPPORTED FIRMWARE VERSIONS:
Upgrading beyond these versions or using other versions may brick your switch and require a recovery procedure!
Name / Types | Version |
SwitchX / SwitchX-2 PowerPC (SX6036, SX6012, SX1016, etc...) | 3.6.8012 (no newer build for PowerPC available) |
SwitchX / SwitchX-2 x86 (SX6710, SX1410, etc...) | 3.6.8012 (upgrading beyond WILL brick your switches BIOS) |
Spectrum x86 (SN2100, SN2700, SN2010, etc) | 3.10.4XXX (DO NOT upgrade to 3.10.5000, 3.10.6004 etc.) |
SwitchIB (SB7700) | 3.9.3124 |
As I don't have a lot of time right now (but want to get this done before I forget it), I'll just start with some basics
Overview of current Mellanox switch series that may be interesting for homelabs:
- Mellanox IS5000 Series: Old, Infiniband only switches. They should generally be avoided:
- Mellanox SX Series: 40G / 56G switches with Switch-X / SwitchX-2 chip. They come in different flavours (unmanaged / managed, full width, half width, etc... TODO: Full Model Listing). They are very energy efficient (TODO: Exact numbers). Most of them are PowerPC, but some are x86
Managed SX Series switches can do VPI - That means you can have some ports do Infiniband, some Ethernet at the same time and even use integrated IPoIB gateway functionality.
Highest supported firmware version (for managed switches): 3.6.8012
DO NOT try to update to a version beyond that on x86 switches - It may brick your switch by automatically doing a bios update that prevents the switch ASIC from being detected!
- Mellanox SB77XX/78XX series: 100G Infiniband switches with Switch-IB / Switch-IB 2 chip (no VPI like SX Series). x86 control plane, highest supported version: 3.9.3124
Very power efficient! SB7700 needs only 53W in IDLE (one PSU)
- Mellanox SN2XXX series: 100G Ethernet switches with Spectrum chip (no VPI like SX series). x86 control plane, highest supported version: ?? Apparently, it's 3.10.4100 (LTS)
Very power efficient. SN2700 needs only 51W in IDLE (one PSU), SN2100 about 35W iirc (after the fans have spun down)
They come in different flavours, but generally you have:
- SN2700: 19" 32x100G, Celeron 1047U, mSATA SSD (TODO: Expand)
- SN2100: 9.5" (half-width), Atom C2XXX (possibly affected by AVR54 bug?), M.2 SATA SSD
... and more (TODO: Full model listing)
Tips & Tricks for working with those switches:
- SN2XXX / SB7XXX series: Replace your SSDs and make backups! The original Innodisk 3ME SSDs are prone to failure (one died while I was taking an image).
You can use whatever mSATA (apparently all except SN2100) or M.2 SATA (apparently only SN2100) you want. My go-to model are Transcend 452T2 (e.g. TS128GMSA452T2)
- If you get an SN2XXX with ONIE, Cumulus or no OS, you can easily flash it to ONYX / MLNX-OS by taking a good MLNX-OS / ONYX image from another switch and patching the embedded database to the correct model number, number of ports, MAC Addresses, etc... (TODO: Guide)
TODO: Expand
Last edited: