Mellanox Switches - Tips & Tricks

[NablaSquaredG]

So to be honest I don't know, but idea for the fix is the same for all of them, so should be possible to apply even with minimal soldering skills.

I don’t think guesswork is good advice here.

Is it possible? Maybe.
Any resources how? Nope.

Perfect. So this means the switch is still usable and it'll get some updates/patches right?

Yes. Mellanox SN2000 also have mainline Linux support and SONiC support.

Onyx 3.6.600

Can you access the switch? You might want to enter a root shell (I can provide a key) and check the CPU Revision.

I see `Rev: A4`

For all Sn2000 I‘ve seen, Rev A meant potentially AVR54 bug, Rev B meant fixed CPU

 

[Rand__]

For all Sn2000 I‘ve seen, Rev A meant potentially AVR54 bug, Rev B meant fixed CPU

Don't you mean Rev C ?

 

[i386]

Someone is trying to sell me a Mellanox SN2010 for 750 euros, and I was wondering if that’s a good price for the switch nowadays. I checked the manufacturer date, and it’s 2018-05-17.

Is the seller from france by any chance?
I've seen an ebay listing for a decent price but backed down due the possibility of the avr54 problem (and front to back airflow )

 

[i386]

This isn't correct.

The terminology is described in the Note at Cable Installation

• C2P = connector to power (ports to PSU flow); AFO (air flow out of chassis [from PSU's perspective]) in Juniper nomenclature; no suffix for most other vendors
• P2C = power to connector (PSU to port flow); AFI (air flow into chassis [from PSU's perspective]) in Juniper nomenclature; -R for most other vendors

You made be think that my post was wrong so I looked again through the different pdfs, sku numbers and replacement parts. (The mellanox documentation for airflow direction sucks...)
It seems mellanox changed the meaning for c2p when they introdcued the sn2xxx series

 

[klui]

They had different meanings in the past?

Datasheets for the SX6012 and it implies the same thing.

https://www.netsolutionworks.com/datasheets/Mellanox_SX6012_SpecSheet.pdf at the bottom: "*P2C is connector side outlet, C2P is connector side inlet."

But it's important to know what Mellanox defines for "connector." At https://content.etilize.com/User-Manual/1027114703.pdf (SwitchX-2 12 Port InfiniBand Switch System Hardware User Manual), page 12 shows "Connector Side View of the Switch."

I personally feel the nomenclature is not very good. In fact, the SX6012's datasheet shows "connector airflow in" SKUs have an "R" suffix. An R basically means reverse flow--PSU to port flow--for many other vendors.

 

[Cheburashka]

Hello, thank you for this wonderful thread. A ton of amazing information that is very helpful to noobs like me.

I am working on a small proof of concept Proxmox hyperconverged setup in my office. This will be the first time I'm working with Mellanox or anything with InfiniBand.

I would like to explore RDMA and it seems Mellanox is a good option for me.

I came across the following switch, MSX1024B-1BFS, which is interesting as it gives me 48-ports 10GbE (which I can use in my test environment but also 12-ports QSFP 40/56GbE but after reading the Mellanox document, it states "Ideal ToR/Core switch using 48 10GbE from server and 12 40/56GbE to aggregation layer". So I am unsure if the 12 QSFP ports can be used for host connectivity? I don't see why not but maybe there is some kind of functionality that is missing from these ports that would benefit server connectivity.

In the event that the MSX1024B-1BFS isn't ideal to use, I'm fine with the typical SX6036 switch.

 

[NablaSquaredG]

So I am unsure if the 12 QSFP ports can be used for host connectivity?

Sure they can.

 

[Cheburashka]

Sure they can.

Okay, thank you. Anyways, back to further researching. Thanks again.

 

[lllllmmm]

Is there anyone who can upload Onyx 3.10.4504? Many thanks.

 

[Cheburashka]

Sure they can.

Thank you,

The switch arrived yesterday and I've started to explore the web and CLI. I was able to update it to the latest version, thanks to these forums.

I do have a question about mgmt0 and mgmt1. These are defined as out of band ports, which I don't plan to use at this time. Normally I manage my HP/Aruba switches from a management VLAN IP. (IE VLAN 32 = 192.168.32.0/24).

The Nvidia documents speak that I can set In-Band management interface this way, but as this switch only has two RJ45 ports and the rest SFP/QSFP, is there any way I can have in-band functionality for these two ports?

I believe OOB ports have their own routing table. My function for this Mellanox switch is to be used for data replication on a Proxmox hyperconverged setup (proof of concept).

I'm currently connected to the management ports to review and configure it via DHCP lease.

Thank you,

 

[NablaSquaredG]

The Nvidia documents speak that I can set In-Band management interface this way, but as this switch only has two RJ45 ports and the rest SFP/QSFP, is there any way I can have in-band functionality for these two ports?

I believe OOB ports have their own routing table. My function for this Mellanox switch is to be used for data replication on a Proxmox hyperconverged setup (proof of concept).

I don't exactly see your problem.

In-Band vs Out-Of-Band is just which ports you are using. If you manage the switch via the normal ports which also pass data, it's in band. If you manage it via the two dedicated ports for mgmt, it's out of band.

And yes, usually it's a sensible thing that the management ports are in a separate VRF.

If you want in band management, just add a VLAN interface (or Loopback or Router Port). The simplest thing will be a VLAN interface. Once that VLAN interface is up and running, you can reach the management plane through its IP.

 

[Cheburashka]

I don't exactly see your problem.

In-Band vs Out-Of-Band is just which ports you are using. If you manage the switch via the normal ports which also pass data, it's in band. If you manage it via the two dedicated ports for mgmt, it's out of band.

And yes, usually it's a sensible thing that the management ports are in a separate VRF.

If you want in band management, just add a VLAN interface (or Loopback or Router Port). The simplest thing will be a VLAN interface. Once that VLAN interface is up and running, you can reach the management plane through its IP.

Thank you again, I completely overlooked the VLAN+Looback interface solution that would work for my case.

 

[bradh352]

I've got a home lab and looking to add some mellanox switches (likely a pair of SN2410 as they seem somewhat cost effective on ebay at the moment). My goal is to test out various VXLAN EVPN scenarios. I already have a pair of Dell S5248F switches which seem to be roughly on par with these but use a Broadcom Trident 3 chip instead of Spectrum.

A few things:

• I've read the guide at the top that said to replace the SSD.
  • The SSD model recommended doesn't seem to be available any more, any specific new recommendations or links on where to get it?
  • When replacing the SSD, recommended to just dd from old to new? Or is there a "fresh install" method to just get ONIE loaded? Where would I download ONIE for this device? It said there was a guide linked later in the thread, but I never found it.
• I'd only plan on testing Cumulus Linux and SONiC, no plans on testing Onyx (which I see downloads linked). I can build SONiC myself as per this NVidia guide (Building an NVIDIA Verified SONiC Image | NVIDIA Technical Blog) which is supposedly a fully validated hash. But any idea where I can get an evaluation version of Cumulus that would run on this?

Thanks!
-Brad

 

[juey90]

Can i safely upgrade my SN2010 directly to 3.10.4408 (LTS) from 3.8.2008 or should i update to 3.9.X as an intermediate step ?

 

[lllllmmm]

I know all the previous posts warned me not to upgrade msn2100 (Spectrum x86) to a higher version than 3.10.4XXX. But today I saw in HPE's release log that "Version 3.10.4500 can be upgraded to verion 3.11.4010 and above". Can anyone verify the feasibility of this upgrade? Maybe according to NVIDIA's version branch 3.10.4XXX can be upgraded to 3.11.4XXX?
Here is the link to HPE's release log (section 3.6): https://support.hpe.com/connect/s/s...ctionId=MTX-81feedf99ac7434d&tab=releaseNotes

 

[juey90]

Can i safely upgrade my SN2010 directly to 3.10.4408 (LTS) from 3.8.2008 or should i update to 3.9.X as an intermediate step ?

Ok nevermind, i went from 3.8.2008 straight to 3.9.3124, then to 3.10.2102 and finally to 3.10.4408 without problems.

 

[Civiloid]

Ok nevermind, i went from 3.8.2008 straight to 3.9.3124, then to 3.10.2102 and finally to 3.10.4408 without problems.

I think that is the correct upgrade path. Basically there are reports that people upgraded from 3.9.3124 directly to 3.10.4408 without visible issues, but it is not clear if mellanox/nvidia removed some scripts in 3.10 and there will be some corner cases that would fail if you don't follow recommended upgrade path.

 

[koifish59]

Firstly, just wanted to say this forum and members have provided invaluable info to help getting my SX6036 up and running. Thank you everyone!

Now my issues. I’m having trouble getting WAN fiber from ISP routed through this switch, and i think it’s mostly from incompatible transceivers. All these optics below works a Brocade ICX6650, but on this SX6036 after using the QSFP to SFP+ adapter, it’s either not detected or detected but no traffic passes, after forcing the speed to 1gbps:

- Brocade 10gbe LR 1310nm single-mode
- Foundry 1gbe LX 1310nm single-mode
- 10Gtek 1gbe LX 1310nm single-mode

Are these switches really that picky for optics, or am I missing something from configs? VLAN seems to be correct (access mode for WAN port, trunk mode for all other ports).

 

[i386]

If I remember it correctly single mode/high power optics worked only on certain ports (see https://forums.servethehome.com/ind...-sx6036-lr4-transceivers-does-not-work.35696/)

 

[amlai]

I have an SN2100. I have both a 100GBASE-PSM4 and a 100GBase-SR4 transceiver and would like to do a 4x 10G breakout. I've been able to configure the ports to do the breakout, but I can only get a link speed of 1G and not 10G. I've tried forcing the 10G speed, but the link just won't come up.

I tried moving the SMF and MMF QSFP28 transceivers to my Mikrotik CRS-504-4XQ-IN, and when I do that, they magically auto-negotiate to 10G. Because of that, I'm only using it essentially as a converter.

In fae mode, I ran the following:

mlxlink -d /dev/mst/mt52100_pci_cr0 -p 15 -m

Operational Info
----------------
State                           : Polling
Physical state                  : ETH_AN_FSM_ABILITY_DETECT
Speed                           : N/A
Width                           : N/A
FEC                             : N/A
Loopback Mode                   : No Loopback
Auto Negotiation                : ON

Supported Info
--------------
Enabled Link Speed              : 0x68b5f141 (100G,56G,50G,40G,25G,10G,1G)
Supported Cable Speed           : 0x2024a101 (100G,56G,50G,40G,25G,10G,1G)

Troubleshooting Info
--------------------
Status Opcode                   : 2
Group Opcode                    : PHY FW
Recommendation                  : Negotiation failure

Tool Information
----------------
Firmware Version                : 13.2010.4406
MFT Version                     : mft 4.22.1-11

Module Info
-----------
Identifier                      : QSFP28
Compliance                      : 100G PSM4 Parallel SMF
Cable Technology                : 1310 nm EML
Cable Type                      : Optical Module (separated)
OUI                             : Other
Vendor Name                     : FS
Vendor Part Number              : QSFP28-PIR4-100G
Vendor Serial Number            : S2202328654
Rev                             : 01
Wavelength [nm]                 : 1310
Transfer Distance [m]           : 0
Attenuation (5g,7g,12g) [dB]    : N/A
FW Version                      : N/A
Digital Diagnostic Monitoring   : Yes
Power Class                     : 4.5 W max
CDR RX                          : ON
CDR TX                          : ON
LOS Alarm                       : N/A
Temperature [C]                 : 36 [-10..80]
Voltage [mV]                    : 3286.4 [3000..3600]
Bias Current [mA]               : 35.358 [7..65]
Rx Power Current [dBm]          : -40 [-14..6]
Tx Power Current [dBm]          : 0 [-6..4]

I should probably try not using a breakout and just go straight from QSFP28 to SFP+, but it seems silly to just utilize entire QSFP28 ports for such low speeds. Does anyone have ideas on how to make 10G work when using the breakout?