Beware of EMC switches sold as Mellanox SX6XXX on eBay

[Necrotyr]

Thank you for the steps!
I’ll wait for confirmation from another veteran of the thread, then proceed.

Also, what is that «Gateway » grayed out tab on the WebUI?
Is that for IB stuff (then I do not care) or is that for L3 inter-VLAN traffic/routing (then I care a lot)?

Step 1.4 is not necessary unless you need to run LR4 optics, but the rest seem fine, I don't personally use the rc.local script, as my switch is not located a place noise matters.

 

[shuguet]

Those switches are ultimately going into a proper datacenter, so I didn't plan on changing it either

Thank you, it looks like it's safe to run those EEPROM dump, will do that ASAP and send them to @Stephan!

 

[Necrotyr]

Those switches are ultimately going into a proper datacenter, so I didn't plan on changing it either

Thank you, it looks like it's safe to run those EEPROM dump, will do that ASAP and send them to @Stephan!

The eeprom dumps only read the content, they don't modify anything.

 

[Stephan]

Wouldn't it be a good idea to start a new Mellanox SX Switch megathread (similar to the ICX6610 one)? The knowledge about those switches is spread across this thread and pretty well hidden for newbies.

I suggest hidden is an advantage, e.g. because it will not force anyone's hand vis-a-vis licensing of current models. In case you want to upgrade. I've seen it many times in my life. If something is too public and accessible, it dies. Also I'm a fan of Randy Pausch's Last Lecture's insight that walls are there for a reason: To prove how badly you want something. 60 pages, small wall. ;-) Check out the video on YT, platinum+ content. Also not sure it's time well spent with the drip of cheap SX60xx being as small as it is.

 

[shuguet]

I'm running into something weird and given that you guys seem to know a lot about those switches, I thought maybe someone could point me in the right direction:

I've got 2 SX1012 (unmodded as far as I know) switches upgraded to the latest version available (see below), however one is reporting as a "Mellanox Onyx SX1012" while the other is reporting as a "Mellanox MLNX-OS SX1012".
So far I have not found real differences in behaviour (still waiting on cables, they are delayed, so can't really test anything beside accessible the mgmt GUI/CLI).

The only physical difference I've seen so far is the manufacturing country, on the physical pull-tab at the front below the mgmt/serial port.
One says made in China (== Onyx), the other one says made in Israel (== MLNX-OS).
In the UI I can see the same FW version on both (9.4.5110), but in the System > Inventory section one says HW Revision AA (== MLNX-OS), and the other HW Revision AB (== Onyx).

Has anyone ran into this before?
Any idea how to get both of them to the "Onyx" path/branch? (which seems to be the "newer" one).
Will it impact their capability to form a cluster/use MLAG?

I've got an EEPROM dump of both CPU and Backplate of both switches, if needed.

 

[klui]

While I have no practical experience with Onyx, the PPC branch stopped at 3.6.8012. All newer versions are x86-based.

 

[shuguet]

Yup, thank you!
The next generation(s) is all x86-based, sn2010/sn2100 series.

What I can't explain is the discrepancy between 2 seemingly identical SKUs (except the rev), upgraded with the exact same image file, by me, to 3.6.8012, yet with one showing up as Onyx and the other one still showing MLNX-OS...

I guess I'll have to wait and get my cables to actually start using both and see if they play nice together.
I was hoping for something very uniform since they are ultimately going to be one cluster/pair

 

[Labs]

What I can't explain is the discrepancy between 2 seemingly identical SKUs (except the rev), upgraded with the exact same image file, by me, to 3.6.8012, yet with one showing up as Onyx and the other one still showing MLNX-OS...

From what I saw there are different images for Onyx and MLNX-OS and from my understanding the Onyx one is for the Ethernet only switches and MLNX-OS can also cover Infiniband. I'm NOT 100% sure but when I checked it a few years ago I remember that was the case.

 

[klui]

Looks like it's purely a marketing differentiation. Eth-only switches default to Onyx while VPI switches default to MLNX-OS. The bits are the same as of 3.6.8012.

Can SX1012 Running MLNX_OS be upgraded to Onyx?

Hi Ben, Neo is backwards compatible with MLNX-OS. MLNX Onyx, is just a new name for the MLNX-OS Ethernet branch to keep separation between the Ethernet switches OS and Infiniband switches OS. You can head over to this page, and see all the available OS for your SX1012...

forums.developer.nvidia.com

 

[Stephan]

So whats the difference that makes the web UI show Onyx not MLNX-OS, switch in VPI vs. ethernet mode?

 

[Labs]

So whats the difference that makes the web UI show Onyx not MLNX-OS, switch in VPI vs. ethernet mode?

I don't think so. I guess it has some list inside which checks for example if it is SX6012 or SX1012.
The SX6012 is VPI and SX1012 is Ethernet Only.

 

[klui]

No, if that was the algorithm it would show Onyx for both of @shuguet's switches. Probably one of two things are happening. Either it parses the FW image used or only the switch components are the same but installation is customized for MLNX-OX and Onyx.

 

[Labs]

No, if that was the algorithm it would show Onyx for both of @shuguet's switches. Probably one of two things are happening. Either it parses the FW image used or only the switch components are the same but installation is customized for MLNX-OX and Onyx.

For download there are separated images to download. I guess one of the switches was upgraded to Onyx by previous owner and one was left with MLNX-OS. Interesting why @shuguet's Onyx upgraded switch didn't switch back to MLNX-OS when he used the MLNX-OS image. It seems the MLNX-OS also has some checking mechanism inside and keeps the Onyx parts even if flashed to MLNX-OS.

I will check later the size of MLNX-OS and Onyx images for the same version. For sure the checksum of images is different because I checked them long time ago but I will double check it.

I checked the Onyx and MLNX-OS images for 8012 and they have exactly the same checksum. Only the filename is different...
That is interesting...

Anyway the PPC images are only up to 8012, x86 images were further developed.

 

[NablaSquaredG]

I've recently been tinkering with the SB7700 engineering sample switch and I think the best way to get this thing to run would be to run a full manufacture cycle on this thing

The SX6012 guide doesn't apply unfortunately, because x86 doesn't use u-boot

Does anyone know how to run a full manufacture cycle on the SB7700?
As far as I remember you can't run manufacture.sh on the same disk where the OS is running... Maybe copy the mSATA SSD with "working" MLNX-OS (doesn't actually work, but linux env works) to USB, boot from USB and then run manufacture.sh?

 

[Labs]

I've recently been tinkering with the SB7700 engineering sample switch and I think the best way to get this thing to run would be to run a full manufacture cycle on this thing

The SX6012 guide doesn't apply unfortunately, because x86 doesn't use u-boot

Does anyone know how to run a full manufacture cycle on the SB7700?
As far as I remember you can't run manufacture.sh on the same disk where the OS is running... Maybe copy the mSATA SSD with "working" MLNX-OS (doesn't actually work, but linux env works) to USB, boot from USB and then run manufacture.sh?

Check out the big archive (400MB) from HP site for the SX60xx, the one used to do the manufacture. It has some zip file inside especially for USB and also a readme.txt. See if you can simulate that with the x86 image for SB7700.

Or second option since it's x86, can you do a PXE netboot? Maybe you can use the vmlinuz/rootfs/fdt with PXE so you can start the manufacture process

BTW: can you post some images how it looks inside the SB7700?

 

[NablaSquaredG]

Check out the big archive (400MB) from HP site for the SX60xx, the one used to do the manufacture. It has some zip file inside especially for USB and also a readme.txt. See if you can simulate that with the x86 image for SB7700.

Yes I've looked at that, but it didn't appear two helpful on the first sight

I have played around with a magic script called imgtoiso.sh which is supposed to create an iso image for factory setup from an image (basically exactly what I need!) - but it is outdated beyond repair (zcat doesn't work because they're not compressing with gzip any longer, script depends on output from losetup which seems to have changed ages ago, filesystem of initrd is not ext2 as expected in the script, etc...)
Also, the SX60XX series used u-boot whereas the x86 version doesn't so I doubt you can derive too much useful information from it

Or second option since it's x86, can you do a PXE netboot? Maybe you can use the vmlinuz/rootfs/fdt with PXE so you can start the manufacture process

How would that be an improvement over boot via USB stick?

BTW: can you post some images how it looks inside the SB7700?

I will take some pics later

 

[Labs]

How would that be an improvement over boot via USB stick?

To boot from USB you need to make a bootable USB stick. If you boot from network you normally need 2 files the rootfs and vmlinuz like when you boot a normal Linux distro from network.

What bootmanager does this switch have? New Mellanox x86 switches I saw they run ONIE mostly and from ONIE you can reinstall the image same as factory one.

 

[NablaSquaredG]

To boot from USB you need to make a bootable USB stick. If you boot from network you normally need 2 files the rootfs and vmlinuz like when you boot a normal Linux distro from network.

Yeah but I think that it doesn't make that much of a difference in the end...

What bootmanager does this switch have? New Mellanox x86 switches I saw they run ONIE mostly and from ONIE you can reinstall the image same as factory one.

GRUB - I believe only the SN series runs ONIE, but the SB is an Infiniband switch

 

[jb221]

I hope this isn't a stupid question but I followed one of the guides successfully (I thought) but now when I run "show system capabilities" is shows ib is supported but when I try to make changes to the interfaces it says no valid license. I don't see anything in the guide about the IB license, did I miss something? I'm converting an EMC SX6012 that I need to work as an infiniband switch specifically. I've got the ethernet and shell command licenses installed currently.

 

[Labs]

@dodgy route I have some updates to your guide version 1.12 for tftp commands and EEPROM dumps.
I noticed on my backups that the backplane bin file is 2~3 bytes bigger. Looks ok on the switch but not on laptop.
I think it is a must to have the mode set to binary.

• tftp 192.168.7.77 -m binary -c put fru_backplate.bin

Also good to backup the CPU EEPROM since we are there:

• /opt/tms/bin/mellaggra _read_fru 0 0x50 1000 fru_cpu.bin

CPU EEPROM dump command is the same for all types 6012/18/36

Backplane EEPROM dump seems to be diferent from my tests and looks like this:

For 6012

• /opt/tms/bin/mellaggra _read_fru 1 0x51 1000 fru_backplate.bin

For 6036

• /opt/tms/bin/mellaggra _read_fru 8 0x51 1000 fru_backplate.bin

And also add the tftp of the CPU EEPROM dump:

• tftp 192.168.7.77 -m binary -c put fru_cpu.bin

I know you have a comment that if 1 is not working try 8 but looks easier to just differentiate for switch types and not create confusion.