Mellanox Switches - Tips & Tricks

[NablaSquaredG]

So to be honest I don't know, but idea for the fix is the same for all of them, so should be possible to apply even with minimal soldering skills.

I don’t think guesswork is good advice here.

Is it possible? Maybe.
Any resources how? Nope.

Perfect. So this means the switch is still usable and it'll get some updates/patches right?

Yes. Mellanox SN2000 also have mainline Linux support and SONiC support.

Onyx 3.6.600

Can you access the switch? You might want to enter a root shell (I can provide a key) and check the CPU Revision.

I see `Rev: A4`

For all Sn2000 I‘ve seen, Rev A meant potentially AVR54 bug, Rev B meant fixed CPU

 

[Rand__]

For all Sn2000 I‘ve seen, Rev A meant potentially AVR54 bug, Rev B meant fixed CPU

Don't you mean Rev C ?

 

[i386]

Someone is trying to sell me a Mellanox SN2010 for 750 euros, and I was wondering if that’s a good price for the switch nowadays. I checked the manufacturer date, and it’s 2018-05-17.

Is the seller from france by any chance?
I've seen an ebay listing for a decent price but backed down due the possibility of the avr54 problem (and front to back airflow )

 

[i386]

This isn't correct.

The terminology is described in the Note at Cable Installation

• C2P = connector to power (ports to PSU flow); AFO (air flow out of chassis [from PSU's perspective]) in Juniper nomenclature; no suffix for most other vendors
• P2C = power to connector (PSU to port flow); AFI (air flow into chassis [from PSU's perspective]) in Juniper nomenclature; -R for most other vendors

You made be think that my post was wrong so I looked again through the different pdfs, sku numbers and replacement parts. (The mellanox documentation for airflow direction sucks...)
It seems mellanox changed the meaning for c2p when they introdcued the sn2xxx series

 

[klui]

They had different meanings in the past?

Datasheets for the SX6012 and it implies the same thing.

https://www.netsolutionworks.com/datasheets/Mellanox_SX6012_SpecSheet.pdf at the bottom: "*P2C is connector side outlet, C2P is connector side inlet."

But it's important to know what Mellanox defines for "connector." At https://content.etilize.com/User-Manual/1027114703.pdf (SwitchX-2 12 Port InfiniBand Switch System Hardware User Manual), page 12 shows "Connector Side View of the Switch."

I personally feel the nomenclature is not very good. In fact, the SX6012's datasheet shows "connector airflow in" SKUs have an "R" suffix. An R basically means reverse flow--PSU to port flow--for many other vendors.

 

[Cheburashka]

Hello, thank you for this wonderful thread. A ton of amazing information that is very helpful to noobs like me.

I am working on a small proof of concept Proxmox hyperconverged setup in my office. This will be the first time I'm working with Mellanox or anything with InfiniBand.

I would like to explore RDMA and it seems Mellanox is a good option for me.

I came across the following switch, MSX1024B-1BFS, which is interesting as it gives me 48-ports 10GbE (which I can use in my test environment but also 12-ports QSFP 40/56GbE but after reading the Mellanox document, it states "Ideal ToR/Core switch using 48 10GbE from server and 12 40/56GbE to aggregation layer". So I am unsure if the 12 QSFP ports can be used for host connectivity? I don't see why not but maybe there is some kind of functionality that is missing from these ports that would benefit server connectivity.

In the event that the MSX1024B-1BFS isn't ideal to use, I'm fine with the typical SX6036 switch.

 

[NablaSquaredG]

So I am unsure if the 12 QSFP ports can be used for host connectivity?

Sure they can.

 

[Cheburashka]

Sure they can.

Okay, thank you. Anyways, back to further researching. Thanks again.

 

[lllllmmm]

Is there anyone who can upload Onyx 3.10.4504? Many thanks.

 

[Cheburashka]

Sure they can.

Thank you,

The switch arrived yesterday and I've started to explore the web and CLI. I was able to update it to the latest version, thanks to these forums.

I do have a question about mgmt0 and mgmt1. These are defined as out of band ports, which I don't plan to use at this time. Normally I manage my HP/Aruba switches from a management VLAN IP. (IE VLAN 32 = 192.168.32.0/24).

The Nvidia documents speak that I can set In-Band management interface this way, but as this switch only has two RJ45 ports and the rest SFP/QSFP, is there any way I can have in-band functionality for these two ports?

I believe OOB ports have their own routing table. My function for this Mellanox switch is to be used for data replication on a Proxmox hyperconverged setup (proof of concept).

I'm currently connected to the management ports to review and configure it via DHCP lease.

Thank you,

 

[NablaSquaredG]

The Nvidia documents speak that I can set In-Band management interface this way, but as this switch only has two RJ45 ports and the rest SFP/QSFP, is there any way I can have in-band functionality for these two ports?

I believe OOB ports have their own routing table. My function for this Mellanox switch is to be used for data replication on a Proxmox hyperconverged setup (proof of concept).

I don't exactly see your problem.

In-Band vs Out-Of-Band is just which ports you are using. If you manage the switch via the normal ports which also pass data, it's in band. If you manage it via the two dedicated ports for mgmt, it's out of band.

And yes, usually it's a sensible thing that the management ports are in a separate VRF.

If you want in band management, just add a VLAN interface (or Loopback or Router Port). The simplest thing will be a VLAN interface. Once that VLAN interface is up and running, you can reach the management plane through its IP.

 

[Cheburashka]

I don't exactly see your problem.

In-Band vs Out-Of-Band is just which ports you are using. If you manage the switch via the normal ports which also pass data, it's in band. If you manage it via the two dedicated ports for mgmt, it's out of band.

And yes, usually it's a sensible thing that the management ports are in a separate VRF.

If you want in band management, just add a VLAN interface (or Loopback or Router Port). The simplest thing will be a VLAN interface. Once that VLAN interface is up and running, you can reach the management plane through its IP.

Thank you again, I completely overlooked the VLAN+Looback interface solution that would work for my case.

 

[bradh352]

I've got a home lab and looking to add some mellanox switches (likely a pair of SN2410 as they seem somewhat cost effective on ebay at the moment). My goal is to test out various VXLAN EVPN scenarios. I already have a pair of Dell S5248F switches which seem to be roughly on par with these but use a Broadcom Trident 3 chip instead of Spectrum.

A few things:

• I've read the guide at the top that said to replace the SSD.
  • The SSD model recommended doesn't seem to be available any more, any specific new recommendations or links on where to get it?
  • When replacing the SSD, recommended to just dd from old to new? Or is there a "fresh install" method to just get ONIE loaded? Where would I download ONIE for this device? It said there was a guide linked later in the thread, but I never found it.
• I'd only plan on testing Cumulus Linux and SONiC, no plans on testing Onyx (which I see downloads linked). I can build SONiC myself as per this NVidia guide (Building an NVIDIA Verified SONiC Image | NVIDIA Technical Blog) which is supposedly a fully validated hash. But any idea where I can get an evaluation version of Cumulus that would run on this?

Thanks!
-Brad