Mellanox Switches - Tips & Tricks

[nasbdh9]

I can confirm that Onyx will be discontinued in the near future (MLNX-OS is not affected, but the eth product will not be supported)
Later support for eth products will be Cumulus Linux and SONiC, or...switchdev

 

[NablaSquaredG]

I can confirm that Onyx will be discontinued in the near future (MLNX-OS is not affected, but the eth product will not be supported)
Later support for eth products will be Cumulus Linux and SONiC, or...switchdev

Yes this is known. Onyx has already been in LTS mode sind some time without major updates.

 

[nasbdh9]

For those interested, here is the Onyx to Cumulus Linux configuration conversion tool.

NVUE Migration Tool

air.nvidia.com

Cumulus Linux and SONiC and even switchdev have lost out-of-the-box webui support for users.
For webui, Cumulus NetQ or various sdn controllers are an alternative, but these require separate deployment and most products are not free or provide community licensing.

 

[NablaSquaredG]

Cumulus Linux and SONiC and even switchdev have lost out-of-the-box webui support for users.

Nobody who needs such a switch should use a WebUI anyway... (Ignoring Netbox with declarative management that automatically deploys to switches - but that's still not a UI for the switch)

 

[Matta]

Can someone confirm or deny following regarding Mellanox SX6012:

1. This should be original Mellanox, correct (black/blue faceplate)?

2. Can this switch be converted from InfiniBand to Ethernet mode ? If yes, how ?

3. Does this switch has web GUI for configuration ? I'm bad at CLI and need only basic stuff.

Thanks in advance !

 

[NablaSquaredG]

Can someone confirm or deny following regarding Mellanox SX6012:

1. This should be original Mellanox, correct (black/blue faceplate)?

2. Can this switch be converted from InfiniBand to Ethernet mode ? If yes, how ?

3. Does this switch has web GUI for configuration ? I'm bad at CLI and need only basic stuff.

Thanks in advance !

1. looks like normal Mellanox, not Dell OEM

2. yes, upgrade to 3.6.8012 and enter the license

license install LK2-EFM_SX-5L11-5M11-5K11-5T11-5U11-5G22-05J2-205N-2205-P220-88A2-L5T5-3MYW-9

Then put switch into right mode / system profile (e.g L3 single or whatever it’s called)
3. yes, but it‘s slow

 

[Matta]

Another question regarding Mellanox SX6012:
- I'm planing to connect this switch and Mikrotik CCR2116-12G-4S+ with QSFP+ to 4xSFP+ breakout cable (Mikrotik has 4 x SFP+ port) and to do interface bonding, to get 40Gbit between those two devices. In theory it should work. Anyone have any experience in similar matter ?
Thanks

 

[NablaSquaredG]

Another question regarding Mellanox SX6012:
- I'm planing to connect this switch and Mikrotik CCR2116-12G-4S+ with QSFP+ to 4xSFP+ breakout cable (Mikrotik has 4 x SFP+ port) and to do interface bonding, to get 40Gbit between those two devices. In theory it should work. Anyone have any experience in similar matter ?
Thanks

Yes it will work. However single stream throughput will be limited to 10Gbit/s due to the way link aggregation works. Total throughout can reach 40Gbit/s but only if you have multiple streams (different source/dest mac or ip or port)

 

[hanisirfan]

May I know if I configured a pair of Mellanox SN 2410 with MLAG, what's the expected behaviour if their IPL is down? I know in fact that MLAG port channels (MPO) will enter the DOWN state. However I'm in still figuring out about the non-MPO ports.

I've done my testing by purposely bringing the IPL down. What I noticed was the non-MPO ports are in the UP state but stops forwarding packets. Is this the normal behaviour for this particular model?

 

[dbCIT]

Hi all,

Successfully upgraded the MSN2410 from Cumulus to ONYX.
yhost [standalone: master] # show version
Product name: Onyx
Product release: 3.10.4408

I did run into a problem with one of them when trying to upgrade and don't know what to do.
I can boot the old software and log in, after the new installation it gives trouble, see logs below.
Anyone has some advice?


NVIDIA Onyx Switch Management
switch-b90672 login: [ 57.080197] pm[5797]: Starting pm: [ OK ]
[ 57.131641] mst[5823]: Starting MST (Mellanox Software Tools) driver set
[ 57.138412] mst[5823]: [warn] mst_pci is already loaded, skipping
[ 57.146252] mst[5823]: [warn] mst_pciconf is already loaded, skipping
[ 57.146522] mst[5823]: Create devices
[ 75.819268] ulogd[7490]: Stopping ulogd: [ OK ]

NVIDIA Onyx Switch Management
myhost login:
NVIDIA Onyx Switch Management
myhost login: admin
Password:
Last login: Fri Jun 22 11:17:15 UTC 2018 on ttyS0
Number of total successful connections since last 1 days: 9

NVIDIA Switch

System is initializing!
This may take a few minutes


Modules are being configured
[ 109.939942] pm[7478]: Stopping pm: [ OK ]
% The connection to management subsystem closed during initialization, exiting CLI.[ 112.092341] sx_low_level_if[8183]: Stopping sx_low_level_if

NVIDIA Onyx Switch Management
myhost login: [ 115.898012] sx_core: Failed to access MTECR, status=4
[ 115.903151] sx_core 0000:03:00.0: Failed to initialize temperature monitoring
[ 115.930482] iounmap: bad address 000000002ac42b03
[ 115.936931] pci_register_driver() returned successfully but probe function failed
[ 115.944561] PCI probe failed, aborting (err=-5)
[ 116.565925] iss-nvram-mac[8484]: Stopping iss-nvram-mac
[ 116.843578] arp_responder[8518]: Stopping arp responder
[ 116.931589] ndisc_ns[8538]: Stopping ndisc ns
[ 117.020864] arp_update[8558]: Stopping arp update
[ 117.179627] mst[8578]: Stopping MST (Mellanox Software Tools) driver set
[ 119.394957] pm[9396]: Starting pm: [ OK ]

 

[NablaSquaredG]

I did run into a problem with one of them when trying to upgrade and don't know what to do.
I can boot the old software and log in, after the new installation it gives trouble, see logs below.
Anyone has some advice?

What is the old version?

Fun fact: I‘m migrating everything from Onyx to Cumulus. Cumulus rocks

 

[dbCIT]

Version summary: X86_64 3.9.2006 2020-12-02 16:59:03 x86_64

Interesting, for this project we need ONYX though

 

[NablaSquaredG]

Version summary: X86_64 3.9.2006 2020-12-02 16:59:03 x86_64

Hm that‘s weird. Can you try letting the 3.9 fully boot and send the output of show asic-version?

Interesting, for this project we need ONYX though

Interesting - what can Onyx do what Cumulus can‘t?

 

[dbCIT]

Show asic-version


myhost [standalone: master] # show asic-version
------------------------------------------------------------------
Module Device Version
------------------------------------------------------------------
MGMT SPC 13.2008.2110

myhost [standalone: master] #

Does this tell you something?


I'll check why we really need ONYX for the project, I offered to use Cumulus but was refused and didn't ask further.
I'm responsible for the Cisco networking and was asked to help with the testing during his holiday.

 

[NablaSquaredG]

Does this tell you something?

Try to update the 3.9 partition to 3.9.3124, that might work.

There are some incompatibilities with ASIC Firmware versions etc…

 

[dbCIT]

Aha, I only have 3.9.3202...
Would it be possible to help me with 3.9.3124?

Thanks for the help so far

 

[NablaSquaredG]

Aha, I only have 3.9.3202...
Would it be possible to help me with 3.9.3124?

Thanks for the help so far

I‘ll see what I can do once I‘m back home in an hour or so

 

[Maddox]

What is the old version?

Fun fact: I‘m migrating everything from Onyx to Cumulus. Cumulus rocks

Interesting, can you share a Cumulus image so we can test it as well ?

 

[dbCIT]

I‘ll see what I can do once I‘m back home in an hour or so

Worked out by using 3.9.3202, thanks for the help.

 

[NablaSquaredG]

FWIW, 3.9.3124 can be downloaded here: https://content.mellanox.com/Software/image-X86_64-3.9.3124.img