Beware of EMC switches sold as Mellanox SX6XXX on eBay

[Rand__]

Brilliant, thanks. Now only need another switch to give it a whirl, but it seems supplies (at affordable prices) have dried up

 

[dodgy route]

Brilliant, thanks. Now only need another switch to give it a whirl, but it seems supplies (at affordable prices) have dried up

Hopefully a brave soul with an old conversion can chime in, particularly if that switch is "spare". Believe the new process should work on any switch.
However I would like to see the output out of the MGMT "backplate" FRU export from an early conversion. I dont think this part is overwritten during the old conversion but worth checking

 

[Rand__]

Well I do have one as mentioned but its in use, so not really keen on mucking around with it at this point

 

[klui]

Hopefully a brave soul with an old conversion can chime in, particularly if that switch is "spare". Believe the new process should work on any switch.
However I would like to see the output out of the MGMT "backplate" FRU export from an early conversion. I dont think this part is overwritten during the old conversion but worth checking

Have you tried it on a switch that has been previous converted by this thread's OP? I will try it on my oldest one this weekend if I have spare cycles. I spent the past two weeks pulling fiber and installing recessed lighting w/ the mandatory sheetrock surgery, drilling holes through headers, and fishing electrical wiring with more to do this weekend. Lighting/house upgrades that have been put off for around a decade.

 

[andvalb]

Have you tried it on a switch that has been previously converted by this thread's OP? I will try it on my oldest one this weekend if I have spare cycles. I spent the past two weeks pulling fiber and installing recessed lighting w/ the mandatory sheetrock surgery, drilling holes through headers, and fishing electrical wiring with more to do this weekend. Lighting/house upgrades that have been put off for around a decade.

I tried. No problem there.
Process simpler than described in the guide.
No need to update the firmware of the switch.
Just need to update fru using a script made by SGS and run manufacture from the TFTP as described in the guide (or use SolarWind TFTP server - it is simpler).
Also as mentioned in the guide you need a standalone HTTP server - I have used an available OpenWrt virtual router - just placed a link's to the image files to the www directory while image files have been located in the TFTP folder of the router alongside with the manufacture (mlnx460ex) folder and files.
Then just run an mfg process and sequential update process by the image fetch/install CLI commands.
Images can be installed with the maximum difference by the 2 in the minor version number.
I.e 3.2 -> 3.4 -> 3.6 . But better to update the bootloader is to use the update sequence that I am described above in the thread. The downgrade is possible but configuration backup is recommended. Maybe an update process can be additionally simplified - by updating the content of the manufacture image files to the versions from the latest image. But this requires an additional investigation.
From the hardware side - if you change loud original fans to one of the low noise and cheap alternatives ("AVC DS04020R12MP 4020" from ali?)
(Note that fan headers have different pinouts and need to be changed).
You need to change the heatsink of the ppc460 CPU package - lower airspeed requires more needles on it.
This also can be fixed by the heatsink from the old GPU cards (like Riva TNT and so on)

 

[dodgy route]

Well I do have one as mentioned but its in use, so not really keen on mucking around with it at this point

Yeah, I'd definitely leave it to someone that has a more spare unit first myself

Have you tried it on a switch that has been previous converted by this thread's OP? I will try it on my oldest one this weekend if I have spare cycles. I spent the past two weeks pulling fiber and installing recessed lighting w/ the mandatory sheetrock surgery, drilling holes through headers, and fishing electrical wiring with more to do this weekend. Lighting/house upgrades that have been put off for around a decade.

Nice one! As mentioned believe it should work with any conversion from past but some steps may be different. like the firmware update, unless you want to revert back to the 3.2.x firmware and let it update from each update, but suspect the old conversion firmware will probably be sufficient.

EDIT: My document is to manufacture an EMC switch from scratch, but should work fine for previously converted switches as well

 

[Necrotyr]

@dodgy route would the guide also work for an SX6018? I have one spare lying around.

If not I could replace my 6012 with the 6018 temporarily, and try.

Both 6012 and 6018 are previously converted.

 

[dodgy route]

@dodgy route would the guide also work for an SX6018? I have one spare lying around.

If not I could replace my 6012 with the 6018 temporarily, and try.

Both 6012 and 6018 are previously converted.

It should, the only thing is the FRU conversion, I dont know firstly how it is on a previously converted SX6012, or SX6018/SX6036.
I could possibly easily determine however, are you able to send me the MGMT "backplate" FRU bin file from your SX6018 so I can compare with my ones. I reckon they will be same just the identifier difference for each switch type. So if thats the case this may be a test and then also update to the doco pending the test

 

[Mithril]

Damn, I finally got my holiday bonus, and it looks like there are no SX6012/SX6018 on Ebay for a reasonable price

 

[neggles]

Have you tried it on a switch that has been previous converted by this thread's OP? I will try it on my oldest one this weekend if I have spare cycles. I spent the past two weeks pulling fiber and installing recessed lighting w/ the mandatory sheetrock surgery, drilling holes through headers, and fishing electrical wiring with more to do this weekend. Lighting/house upgrades that have been put off for around a decade.

I have two EMC SX6012 which I ran through the entire old conversion process on (several times trying to get it right - my file package and slightly revised guide is linked somewhere in this thread, I forget where) but never got around to actually using.

I ran both of them through essentially the same conversion process given in the new doc - I did it myself, manually, several weeks ago before the new doc existed - and it went completely fine.

This whole process is essentially just tweaking some IDs in the onboard EEPROMs so the switch is correctly identified as an SX6012, then running through the same process Mellanox would've run at the factory to install MLNX-OS if these hadn't been EMC OEM units. That 'manufacturing install' process is intended to be run on a switch with empty or corrupt NAND, one of the first things it does is a complete clean wipe of all four OS/data partitions - it doesn't matter what's already on there.

@dodgy route one thing I should add - you don't have to apply the 3.4.0012 image then upgrade to latest, you can feed the 3.6.8012 image to /sbin/manufacture.sh and it will install just fine. The actual manufacturing environment is not version-specific - you can see what I did to get from my old modified environment to the new one in this copy of my notes from when I did it

[Edit] Oh, and, fun fact - that license generation secret is unchanged even in the latest version of ONYX for the SN2/3/4000 series...

 

[dodgy route]

I have two EMC SX6012 which I ran through the entire old conversion process on (several times trying to get it right - my file package and slightly revised guide is linked somewhere in this thread, I forget where) but never got around to actually using.

I ran both of them through essentially the same conversion process given in the new doc - I did it myself, manually, several weeks ago before the new doc existed - and it went completely fine.

This whole process is essentially just tweaking some IDs in the onboard EEPROMs so the switch is correctly identified as an SX6012, then running through the same process Mellanox would've run at the factory to install MLNX-OS if these hadn't been EMC OEM units. That 'manufacturing install' process is intended to be run on a switch with empty or corrupt NAND, one of the first things it does is a complete clean wipe of all four OS/data partitions - it doesn't matter what's already on there.

@dodgy route one thing I should add - you don't have to apply the 3.4.0012 image then upgrade to latest, you can feed the 3.6.8012 image to /sbin/manufacture.sh and it will install just fine. The actual manufacturing environment is not version-specific - you can see what I did to get from my old modified environment to the new one in this copy of my notes from when I did it

[Edit] Oh, and, fun fact - that license generation secret is unchanged even in the latest version of ONYX for the SN2/3/4000 series...

Absolutely agree with everything said here, it's the best option to redo it from scratch as per factory and have full functionality.

However I tried to manufacture directly with 3.6 (any) and didn't have much luck
Even after fixing the space issues the scripts were complaining about.

I reckon there must have been some minor changes in later manufacture.sh or other scripts like writeimage.sh but didn't bother. In essence yes. It doesnt matter which version you do the process is the same

 

[neggles]

Absolutely agree with everything said here, it's the best option to redo it from scratch as per factory and have full functionality.

However I tried to manufacture directly with 3.6 (any) and didn't have much luck
Even after fixing the space issues the scripts were complaining about.

I reckon there must have been some minor changes in later manufacture.sh or other scripts like writeimage.sh but didn't bother. In essence yes. It doesnt matter which version you do the process is the same

I do recall I had to do something to make it not throw a hissyfit - I used a different set of switches to what you use in the guide:

Guide calls /sbin/manufacture.sh -a -m ppc -B -u <url> which means these options:

-a (auto): do not prompt for configuration and layout choices; take defaults from model definition
-m MODEL: system to manufacture (i.e. echo100)
-B: force install of bootmgr

I ran manufacture.sh -v -v -t -a -m ppc -u <url> which is these options (ignoring the two -v flags for higher debug level):

-t: don't use tmpfs for a working area during manufacture  will attempt to use space in /var
-a (auto): do not prompt for configuration and layout choices; take defaults from model definition
-m MODEL: system to manufacture (i.e. echo100)

I believe the issue I ran into was a lack of ramdisk space when deploying the 3.6.8012 image (as it's quite a bit bigger I think) - so I used the -t switch to make it use the NAND as a temporary area instead. This makes the whole process take absolutely forever - I think it was almost 2 hours - because the NAND is so slow, but it doesn't run out of space. You could probably just tweak the ramdisk size when booting to manufacture mode instead, but I wasn't sure how big it needed to be & I didn't want to reboot repeatedly.

I also didn't bother to force-reinstall u-boot (bootmgr, -B flag) as I'd already flashed the mellanox u-boot manually earlier on.

 

[Necrotyr]

It should, the only thing is the FRU conversion, I dont know firstly how it is on a previously converted SX6012, or SX6018/SX6036.
I could possibly easily determine however, are you able to send me the MGMT "backplate" FRU bin file from your SX6018 so I can compare with my ones. I reckon they will be same just the identifier difference for each switch type. So if thats the case this may be a test and then also update to the doco pending the test

I've sent you a pm with the fru backplane file, if you could have a look

 

[klui]

After this manufacture process does the shell allow you to login as root so you could remount the filesystem in rw mode to modify fan RPM per https://forums.servethehome.com/ind...-as-mellanox-sx6xxx-on-ebay.10786/post-282522 ?

 

[andvalb]

After this manufacture process does the shell allow you to login as root so you could remount the filesystem in rw mode to modify fan RPM per https://forums.servethehome.com/ind...-as-mellanox-sx6xxx-on-ebay.10786/post-282522 ?

_shell command is present after license installation.

 

[Necrotyr]

_shell command is present after license installation.

Which means you can manually change fan speed, but can't edit RC.local (to make it modify fan speed at boot), as it's on root fs, which is read-only.

Scratch that, I just tried remounting root from _shell, was able to do it and modify rc.local, my changes stayed after reboot, so it should work.

@klui seems it should work:

enter _shell, mount root as read-write

mount -o remount, / rw

and edit rc.local.

After reload of switch root is back to read-only, but changes stay.

Bear in mind I'm testing on 3.4.0012 as I haven't completed the guide yet, as I'm waiting on dodgy route for the FRU changes on an SX6018.

 

[dodgy route]

I do recall I had to do something to make it not throw a hissyfit - I used a different set of switches to what you use in the guide:

I believe the issue I ran into was a lack of ramdisk space when deploying the 3.6.8012 image (as it's quite a bit bigger I think) - so I used the -t switch to make it use the NAND as a temporary area instead. This makes the whole process take absolutely forever - I think it was almost 2 hours - because the NAND is so slow, but it doesn't run out of space. You could probably just tweak the ramdisk size when booting to manufacture mode instead, but I wasn't sure how big it needed to be & I didn't want to reboot repeatedly.

This is very interesting, I did see this stuff in the scripts but decided not to play with it, in all honesty 2 hours is nothing, it takes 1 hour to upgrade both partitions per each image
However my testing is done for this, I am happily using my switch in home "production" so starting from scratch activity is no longer possible for me

I also didn't bother to force-reinstall u-boot (bootmgr, -B flag) as I'd already flashed the mellanox u-boot manually earlier on.

Yes! This is only required once, even if you manufacture the switch multiple times

I've sent you a pm with the fru backplane file, if you could have a look

We have been having a PM blast, hopefully testing goes well!

Which means you can manually change fan speed, but can't edit RC.local (to make it modify fan speed at boot), as it's on root fs, which is read-only.

Scratch that, I just tried remounting root from _shell, was able to do it and modify rc.local, my changes stayed after reboot, so it should work.

@klui seems it should work:

enter _shell, mount root as read-write

mount -o remount, / rw

and edit rc.local.

After reload of switch root is back to read-only, but changes stay.

Bear in mind I'm testing on 3.4.0012 as I haven't completed the guide yet, as I'm waiting on dodgy route for the FRU changes on an SX6018.

I havent played with the fan mods stuff yet but it should all be doable from _shell easily as you pointed out, its rather warm down under and where my switch is currently 40% fan speed is ideal for noise/temperature wise. Also as mentioned FRU info updated, yay!

Pending this working out with the SX6018, I would be keen to hear from someone with an EMC SX6036, converted or not, but preferably able to test...

 

[RedX1]

Hello



I do not have any EMC Mellanox SX6XXX switches, but I have taken several IBM and HP SX1036 and SX6036 Switches through the upgrade process.

The upgrade process using the Web-GUI for the HP images takes around 25 mins to upload the image and then another 25 mins to fully install it.

The upgrade from the HP 3.6.8010 to the Lenovo 3.6.8012 image is not as straightforward. The process is very slow and will take almost 120 mins. (There is a counter).

If you are installing multiple images you will need to remove the old web-images, otherwise the process will fail.

Please see posts 20 and 24 from this thread.
https://forums.servethehome.com/index.php?threads/us-mellanox-sx6036-200.31513/


I hope this helps.



Good luck.





RedX1

 

[i386]

The process is very slow and will take almost 120 mins.

Weird, my hpe branded 6036 fninished in less than 30minutes (including the reboot)

 

[dodgy route]

Yes same here, no troubles with 8012