Thanks that helps!
I am going to be building an Infiniband network for storage, and that is good to know.
Mellanox Switches - Tips & Tricks
- Thread starter NablaSquaredG
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
I am going to be building an Infiniband network for storage, and that is good to know.
fyi, sometimes you have to do the intermediate step.
The last switch I had, I tried to go from 3.8.xx to 3.10.xx - no luck!
I've also had a switch where I went straight to 3.10.xx
You would think so.
In the end, I gave up, did the intermediate step & moved on.
I might have PM'd you when I was having those issues!
Alright, my experiences from flashing 4 SN2700 (Rev B7):
No problems with baudrate. BIOS 0ABZS017_02.02.002 was already installed, was upgraded to 0ABZS017_02.02.004.
The
The ONIE
Remember to use legacy boot, as UEFI boot will give you a
PSA: 3.9.3202 seems to be a big buggy, if you don't log in quick enough to run the initial setup it will report that it has lost the connection to the control plane. Also if you plug in the management LAN when you haven't run through initial setup will lockup the switch and requires a reboot...
No problems with baudrate. BIOS 0ABZS017_02.02.002 was already installed, was upgraded to 0ABZS017_02.02.004.
The
X86_64-3.10.4206-installer.bin didn't work for me at all. I couldn't make it work in any combination.The ONIE
2020.11-5.3.0005 in legacy boot + X86_64-3.9.3202-installer.bin works pretty well.Remember to use legacy boot, as UEFI boot will give you a
error: disk (hd0,4) not found error for some reason.PSA: 3.9.3202 seems to be a big buggy, if you don't log in quick enough to run the initial setup it will report that it has lost the connection to the control plane. Also if you plug in the management LAN when you haven't run through initial setup will lockup the switch and requires a reboot...
Nope. I tried a couple of combinations (new / old ONIE, 3.9 / 3.10 installer) but none worked.
115200 everywhere.
Performance question...
On ConnectX3 non-Pro, Linux, ntttcp or nuttcp, 4 streams sender, 4 streams receiver (same box), IP namespace loopback via FDR QSFP+ DAC cable, link speed set to 56 Gbps, I can barely get over 40 Gbps on a 8259CL CPU on an X11SPL-F board. Typical speed is around 39-40 Gbps over the course of a one hour run.
Tried disabling all sleep states of CPU (barely an effect, 1 Gbps or so), I disabled memory init/teardown via boot init_on_alloc=0 init_on_free=0 (a handful of Gbps more), MTU set to 9000 (nothing), various sysctl tweaks (nothing), other tools than ntttcp or nuttcp like iperf/iperf3 way worse (3-10 Gbps less in aggregate), adaptive-rx off rx-usecs 0 tx-frames 64 (a few Gbps more, at best).
What limit exactly am I hitting here?
On ConnectX3 non-Pro, Linux, ntttcp or nuttcp, 4 streams sender, 4 streams receiver (same box), IP namespace loopback via FDR QSFP+ DAC cable, link speed set to 56 Gbps, I can barely get over 40 Gbps on a 8259CL CPU on an X11SPL-F board. Typical speed is around 39-40 Gbps over the course of a one hour run.
Tried disabling all sleep states of CPU (barely an effect, 1 Gbps or so), I disabled memory init/teardown via boot init_on_alloc=0 init_on_free=0 (a handful of Gbps more), MTU set to 9000 (nothing), various sysctl tweaks (nothing), other tools than ntttcp or nuttcp like iperf/iperf3 way worse (3-10 Gbps less in aggregate), adaptive-rx off rx-usecs 0 tx-frames 64 (a few Gbps more, at best).
What limit exactly am I hitting here?
Interesting. Seems like this does the trick! ONIE
2020.11-5.3.0005 + X86_64-3.10.4206-installer.bin is now working smoothly.Glad it works, I also have two switches to prepare, is it possible to have access to the installation file X86_64-3.10.4206-installer.bin?
I only got it from another member in a PM, so I can't really share it. Maybe they're going to PM you too...
I just purchased an MSX1710-BS2F2 and I need to get past version MLNX-OS 3.4.3002 for this switch to enable 56GbE. Could someone share a newer firmware for this?
Thanks
Thanks
So, I missed the note on not upgrading SX models past 3.6. So I upgraded to 3.9 and things went awry.
I did not equate MSX == SX . As expected the ASIC is no longer available.
The main issue was attempting to find any version that was less than 3.6 and greater than 3.4.12 .
So for now I think this will look good on my server room wall.
Good thing I got a decent deal on this, only paid $250.00 for the new wall art.
Last edited: