Mellanox Switches - Tips & Tricks

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
F

Freebsd1976

Active Member
Feb 23, 2018
405
73
28
dbCIT said:
Hi all,

Successfully upgraded the MSN2410 from Cumulus to ONYX.
yhost [standalone: master] # show version
Product name: Onyx
Product release: 3.10.4408

I did run into a problem with one of them when trying to upgrade and don't know what to do.
I can boot the old software and log in, after the new installation it gives trouble, see logs below.
Anyone has some advice?


NVIDIA Onyx Switch Management
switch-b90672 login: [ 57.080197] pm[5797]: Starting pm: [ OK ]
[ 57.131641] mst[5823]: Starting MST (Mellanox Software Tools) driver set
[ 57.138412] mst[5823]: [warn] mst_pci is already loaded, skipping
[ 57.146252] mst[5823]: [warn] mst_pciconf is already loaded, skipping
[ 57.146522] mst[5823]: Create devices
[ 75.819268] ulogd[7490]: Stopping ulogd: [ OK ]

NVIDIA Onyx Switch Management
myhost login:
NVIDIA Onyx Switch Management
myhost login: admin
Password:
Last login: Fri Jun 22 11:17:15 UTC 2018 on ttyS0
Number of total successful connections since last 1 days: 9

NVIDIA Switch

System is initializing!
This may take a few minutes


Modules are being configured
[ 109.939942] pm[7478]: Stopping pm: [ OK ]
% The connection to management subsystem closed during initialization, exiting CLI.[ 112.092341] sx_low_level_if[8183]: Stopping sx_low_level_if

NVIDIA Onyx Switch Management
myhost login: [ 115.898012] sx_core: Failed to access MTECR, status=4
[ 115.903151] sx_core 0000:03:00.0: Failed to initialize temperature monitoring
[ 115.930482] iounmap: bad address 000000002ac42b03
[ 115.936931] pci_register_driver() returned successfully but probe function failed
[ 115.944561] PCI probe failed, aborting (err=-5)
[ 116.565925] iss-nvram-mac[8484]: Stopping iss-nvram-mac
[ 116.843578] arp_responder[8518]: Stopping arp responder
[ 116.931589] ndisc_ns[8538]: Stopping ndisc ns
[ 117.020864] arp_update[8558]: Stopping arp update
[ 117.179627] mst[8578]: Stopping MST (Mellanox Software Tools) driver set
[ 119.394957] pm[9396]: Starting pm: [ OK ]
Click to expand...
when upgrade plz read release note , which will tell th upgrade path.
Btw: could you share the 4408 img?
 
B

Bene11660

New Member
Jan 26, 2019
15
6
3
Anyone know if its posssible to convert the HP SN2410bM 10G SFP+ Ports to 25G SFP28?
Im planning to use switchdev/openwrt and wondering if the limitation is software only.
 
B

Bene11660

New Member
Jan 26, 2019
15
6
3
Thank you very much, so the plan would be:
- Buy the switch
- Backup the SSD
- Upgrade ASIC/Firmware through Factory OS
- Install desired OS

Is that correct and should the Port Speeds be unlocked by then?
 
Last edited:
Y

Yunochi

New Member
Jul 12, 2019
11
10
3
Is it worked for SN3XXX? or I need other Image/installer?

Edit: I misunderstood what SPC means. resolved.
 
Last edited:
N

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,729
1,156
113
JBB said:
Have someone the 3.10.4408 image and/or the 3.10.3308 Full Release notes?
Click to expand...
3.10.4408

Code: 
3.10.44xx
3748676
Third-Party Modules
Some third-party modules are causing false-alarmed interrupts during SERDES tuning, which overload the ASIC and causing delay in its response.
3.10.4404
Code: 
3527932
VLAN
After removing the VLAN interface using the IPL configuration and rebooting the switch, errors are seen in the log.

3571204
LACP
When combining LACP fast rate and LACP slow rate on different devices, the LACP link flapping occurs.

3510518
Cables
Tx bias current shows N/A on some cables.

3700976
LDAP
Fixed the "group-attribute/group-dn" LDAP configurations. If the "group-dn" is set, a user must be a member of this group or the user will not be authorized to log in as the membership of group is set by the group-attribute.

3565862
IGMP
IGMP snooping sends Source and Group Specific queries as a response to the received IGMPv3 Current-State-Record Membership report.
 
N

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,729
1,156
113
Does anyone have an SN2410 and could please run sudo mlxfwmanager and decode-syseeprom (Cumulus)? I have one with messed up ASIC EEPROM and I'm not sure which PN to flash...
 
A

aprrodrigues

New Member
Jun 7, 2024
1
0
1
Hello all,

I have a SN2700 with Sonic Linux and i want to install Onyx, but i download X86_64-3.9.3202-installer.bin and doesn't work.

Anyone can help?
 
N

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,729
1,156
113
FYI, if you ever need to recover the SMBIOS (it's important, as Cumulus uses the info), the following command might help

Example for an SN2410:
Code: 
./amidelnx_64 /dpc /SM "Mellanox Technologies Ltd." /SP MSN2410 /SV A4 /SS <Switch Serial Number> /SK MSN2410 /SF "Mellanox Switch family" /BM "Mellanox Technologies Ltd." /BP SA002774 /BV A3 /BS <Management Board Serial Number> /CM "Mellanox Technologies Ltd." /CV A4 /CS <Switch Serial Number> /CSK MSN2410
Note that you might have to replace /BP SA002774 /BV A3 (info about mgmt board) with another SKU (usually SA000874) + Rev depending on which switch you have.

Also, the procedure is like:
1. Update SMBIOS
2. Update BIOS (because dmidecode will complain about some amount of bytes that doesn't match, or something like Wrong DMI structures count: 77 announced, only 75 decoded.)
3. Double Check that SMBIOS info is correct, otherwise update SMBIOS again

A healthy DMI info looks like this:
Code: 
ONIE:/ # dmidecode -t1
# dmidecode 3.1
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.

Handle 0x0001, DMI type 1, 27 bytes
System Information
    Manufacturer: Mellanox Technologies Ltd.
    Product Name: MSN2700
    Version: A4
    Serial Number: MT1745K01647
    UUID: D7F3B8D8-C373-11E7-8000-EC0D9A56CF00
    Wake-up Type: Power Switch
    SKU Number: MSN2700
    Family: Mellanox Switch family

ONIE:/ # dmidecode -t2
# dmidecode 3.1
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
    Manufacturer: Mellanox Technologies Ltd.
    Product Name: SA000874
    Version: A2
    Serial Number: MT1742K03309
    Asset Tag: Not Specified
    Features:
        Board is a hosting board
        Board is replaceable
    Location In Chassis: Not Specified
    Chassis Handle: 0x0003
    Type: Motherboard
    Contained Object Handles: 0

ONIE:/ # dmidecode -t3
# dmidecode 3.1
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.

Handle 0x0003, DMI type 3, 22 bytes
Chassis Information
    Manufacturer: Mellanox Technologies Ltd.
    Type: Rack Mount Chassis
    Lock: Not Present
    Version: A4
    Serial Number: MT1745K01647
    Asset Tag: Not Specified
    Boot-up State: Safe
    Power Supply State: Safe
    Thermal State: Safe
    Security Status: None
    OEM Information: 0x00000000
    Height: Unspecified
    Number Of Power Cords: 1
    Contained Elements: 0
    SKU Number: MSN2700
 
Last edited:
  • Like
Reactions: Freebsd1976
N

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,729
1,156
113
Oh and another warning:

On ONIE based switches, DO NOT modify the System FRU / Chassis FRU EEPROM like you do in the SX6012 conversion guide.
It will break the ONIE checksum and cause a lot of issues.
 
S

stefan77

New Member
Mar 30, 2017
7
0
1
47
Mellanox SX6720 port mode question:

First post of the thread states that the SX series supports both infiniband and ethernet, which is what I look for. So, the SX6720 model seems pretty interesting due the x86 control plane compared to the older models like SX6036. But:

Does anyone know whether the SX6720 model supports both infiniband and ethernet? And if yes, can they be mixed per port?

(documentation is a bit unclear on the topic)
 
You must log in or register to reply here.
Top Bottom