Mellanox Switches - Tips & Tricks

[awedio]

sn2700s are funny, they want 9600, but update them, and new onyx wants 115200... but get into old bios.. yeah 9600

That is so true!

 

[ano]

That is so true!

I thought I had bricked one the first time I did the onie/onyx upgrade.....

 

[dbTH]

sn2700s are funny, they want 9600, but update them, and new onyx wants 115200... but get into old bios.. yeah 9600

It's not just the ONYX software problem. Every switch hardware such as SN2xxx hardware has a revision #. The older Rev. # won't work better than the newer. I would look for the newest one unless the older one is so cheap that you could just use it as an experimental box.

Also the hardware UART chip performance could be degraded over the life time, which also depends how you operate it or whether you have enough sysem cooling

 

[awedio]

Love me some SN2700...

 

[awedio]

USB-C to RJ45 Console Cable
Using this in Windows 11 (driver auto installed itself)

 

[SadoKitten]

Messing around with a SN2700 o/s re-install.
I keep getting this error.
Any ideas?

Error: Partition doesn’t exist.

== Creating ext3 filesystem on /dev/sda10 for VAR

mke2fs: No such device or address while trying to determine filesystem size

*** Could not make filesystem on /dev/sda10

== Cleanup

Mine did that until I replaced the ssd

 

[awedio]

Mine did that until I replaced the ssd

What SSD was in there during the error?

What SSD made the error go away?

 

[awedio]

MELLANOX MCP7F00 30AWG 1M 100GbE to 4X25GbE DAC Splitter Cable "Brand New" for less than $25 each

 

[dbTH]

MELLANOX MCP7F00 30AWG 1M 100GbE to 4X25GbE DAC Splitter Cable "Brand New" for less than $25 each

I got the 2M length cable from the same seller. The 1M length may not be long enough in most situation unless the switch is next to the servers

 

[awedio]

For a noob like myself, how does one get shell access on a SN2700?

 

[NablaSquaredG]

For a noob like myself, how does one get shell access on a SN2700?

Install the shell license LK2-RESTRICTED_CMDS_GEN2-88A1-NEWD-BPNB-1 and use _shell command

 

[SadoKitten]

I know the switch supports 25gbe, but Is a breakout cable the only way to achieve this?

 

[NablaSquaredG]

I know the switch supports 25gbe, but Is a breakout cable the only way to achieve this?

No, you can use a QSA28 adapter (QSFP28->SFP28) or a 100GBase-SR4 transceiver

 

[SadoKitten]

Adapters will be the way to go for us then , we got 25gbe dual port cards in last week

 

[SadoKitten]

Can you have a 25gbe to 100gbe breakout cable go to 2 different servers each with dual 25gbe ports ?

 

[NablaSquaredG]

Can you have a 25gbe to 100gbe breakout cable go to 2 different servers each with dual 25gbe ports ?

yes.
The splitted / breakout port behaves like 4 true separate ports.

Please note that using breakout on a port will block the port below (if you use 4x25G breakout), as the switch supports 64 Ports max.
If you use 2x50G breakout, no ports will be blocked.

 

[awedio]

Here's what a "split" port looks like: (Port 6, below it, is no longer useable).
Port 5 is "split" into 4 separate ports: 5-1, 5-2, 5-3 and 5-4.

 

[dbTH]

yes.
The splitted / breakout port behaves like 4 true separate ports.

Please note that using breakout on a port will block the port below (if you use 4x25G breakout), as the switch supports 64 Ports max.
If you use 2x50G breakout, no ports will be blocked.

I guess because of 64 ports max limitation on Mellanox SN2XXX switches, only the top row 16 QSFP28 ports on SN27xxx switches can be split into 4 SFP28 ports each ( 16x4), thus blocking the remaining ports on bottom row. And SN2100 or SN2010 won't go beyond 64 ports limits even if all QSFP28 ports are split, and so all 100GGbE ports are splittable

This page of the SN2000 switch manual about SN2000 Cable Installation is useful for the reference:

Cable Installation

docs.nvidia.com

 

[awedio]

dd cloning SN2700 image: from 256gb mSATA to a 64gb mSATA.

dd cloning operation is failing, here are the commands I have tried.

sudo dd status=progress if=/dev/sd256 of=/dev/sd64 bs=4096 conv=notrunc,noerror

sudo dd status=progress if=/dev/sd256 of=/dev/sd64 bs=512 count= 69206016

What is the proper command?

 

[Freebsd1976]

anyone has 3.10.4206? installer or update image?
according release notes, this version fix boot i2c init and ntp issue, important for everyone