Mellanox Switches - Tips & Tricks

[mpchris]

Each image file includes an MFG system (you don’t have to use the MFG version that matches the image).

After booting to MFG, run the included manufacture.sh script, this does not require installing ONIE. For switches that use the traditional boot method, please use this installation approach.

Ohh interesting, how do I boot into MFG tho? I see that the img contains the mfg-initrd and mfg-kernel but my brain is too smooth right now to figure out what to do with that :/
When I boot to single user mode I can find the manufacture.sh in /sbin, but I don't think that is the right way to run this script?

 

[Tim111]

Each image file includes an MFG system (you don’t have to use the MFG version that matches the image).

After booting to MFG, run the included manufacture.sh script, this does not require installing ONIE. For switches that use the traditional boot method, please use this installation approach.

Can this be done from single user mode?
He says he can't get to regular shell.

 

[Tim111]

Ohh interesting, how do I boot into MFG tho? I see that the img contains the mfg-initrd and mfg-kernel but my brain is too smooth right now to figure out what to do with that :/
When I boot to single user mode I can find the manufacture.sh in /sbin, but I don't think that is the right way to run this script?

sh ./sbin/manufacture.sh

 

[alexv92]

Hi everyone,

I’m trying to get RoCEv2 over NVMe-oF working through an SN2100 switch, and I’ve hit a wall.
Direct connection (no switch):

rping and nvme discover work perfectly on all hosts.


Through SN2100:

rping -c -a <IP> fails with RDMA_CM_EVENT_UNREACHABLE, error -110
nvme discover times out
BUT: iperf3 works at ~98 Gbps → basic L2/L3 forwarding is fine


Hosts:

Mellanox ConnectX-6 Dx EN (MCX623106AN-CDA_Ax)
Ubuntu 22.04 / 24.04 / 25.04
All ports configured for RoCEv2:
ROCE_CONTROL = ROCE_ENABLE(2)
QOS_TRUST_STATE = TRUST_DSCP(2)
ROCE_RTT_RESP_DSCP = 26, CNP_DSCP = 48
ROCE_CC_PRIO_MASK = 32 (for DSCP 26 → prio 5)
LLDP_NB_* = ALL, DCBX = True
MTU = 9216
IPv4-mapped GID ( ::ffff:10.20.x.x) present



Tested firmware versions (none worked):

3.10.2102
3.10.4006
3.10.4400
3.10.4606
3.10.2102
3.7.1134
3.9.3302


All of them result in the same UNREACHABLE error.

Has anyone successfully run RoCEv2 through an SN2100 switch? If yes, what ONYX version did you use?
Is there a known workaround for this issue in dev builds?
Would Cumulus Linux 5.5+ be a reliable alternative for RoCE/NVMe-oF on SN2100?


Any advice or similar experiences would be greatly appreciated!

 

[mpchris]

sh ./sbin/manufacture.sh

No I meant if single user mode is the right environment to run this script, I generally know how to run scripts
It probably isn't, also the script needs some arguments.

 

[Tim111]

No I meant if single user mode is the right environment to run this script, I generally know how to run scripts
It probably isn't, also the script needs some arguments.

I never ran it nasbdh9 might know what you need but I would think yes to single user mode.