Beware of EMC switches sold as Mellanox SX6XXX on eBay

[larryl79]

Somebody could me tell me / point on a post where can see all models of buyable managed IB-ethernet gateways (even convertible)

Thanks

 

[NablaSquaredG]

Somebody could me tell me / point on a post where can see all models of buyable managed IB-ethernet gateways (even convertible)

SX6036
SX6018
SX6012
SX6720
SX6710

 

[larryl79]

SX6036
SX6018
SX6012
SX6720
SX6710

Thanks for the quick answer.

Which need be converted? all supported by MLX-OS?

 

[Softlution]

This is for anybody that finds this huge thread and want to skip most of the out of date info and just get a clean updated switch



Hello All, Thought I would post my experience successfully converting my EMC SX6018 using the guide here...

https://forums.servethehome.com/index.php?threads/beware-of-emc-switches-sold-as-mellanox-sx6xxx-on-ebay.10786/post-322431

Converting Mellanox EMC switch to SX6012.docx

I originally flashed using the "hacky" guide but this updated guide works much better and gives you a "real" install.

I was able to do the update using a MAC with the built in apache server and tftp server to host the necessary files (It should be equally easy on linux)

How to Use the Built-in TFTP Daemon on Mac OS

WirelessPhreak.com is my personal blog where I post on topics related to Technology, Security, Travel, and other random stuff I like.

www.wirelessphreak.com

Enable The Built-In Web Server In OS X – krypted

You don't need php or anything fancy in apache, just the ability to host files

All the files needed are linked in the above doc.

Once you successfully flash the first image you can just use the web gui to continually update to the next version

I had a few updated versions refuse to start (hung on initializing), when that happened I just booted back to the last version and moved on to the next one and it worked fine. I wonder if I could have jumped directly to the last version but I did all versions except for two.

There is lots of information on this thread about updating fans but here is the best way to control fan speed

/opt/tms/bin/mdreq action /system/chassis/actions/set-fan-speed fan_module string "/FAN/FAN" fan_number int8 1 fan_speed int8 27 set_max uint8 50
/opt/tms/bin/mdreq action /system/chassis/actions/set-fan-speed fan_module string "/FAN/FAN" fan_number int8 2 fan_speed int8 27 set_max uint8 50
/opt/tms/bin/mdreq action /system/chassis/actions/set-fan-speed fan_module string "/FAN/FAN" fan_number int8 3 fan_speed int8 27 set_max uint8 50
/opt/tms/bin/mdreq action /system/chassis/actions/set-fan-speed fan_module string "/FAN/FAN" fan_number int8 4 fan_speed int8 27 set_max uint8 50
/opt/tms/bin/mdreq action /system/chassis/actions/set-fan-speed fan_module string "/PS1/FAN" fan_number int8 1 fan_speed int8 27 set_max uint8 50
/opt/tms/bin/mdreq action /system/chassis/actions/set-fan-speed fan_module string "/PS2/FAN" fan_number int8 1 fan_speed int8 27 set_max uint8 50

you can throw it in the /etc/rc.d/rc.local script to start at boot
To do this you need to login to the switch, enable changes, and run # _shell
check where the current mount point is by running
mount
then you can remount the root partition with the following command
mount -o remount,rw /dev/mtdblock7 /

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local
touch /var/lock/subsys/local

echo "Waiting for clusterd"
COUNT=1
while true
do
    CMD=$(ps aux|grep "/[o]pt/tms/bin/clusterd")
    if [ $? -ne 0 ]; then
        sleep 30
        let COUNT=${COUNT}+1
        if [ $COUNT -gt 10 ]; then
            echo "time out, aborting"
            break
        fi
    else
        sleep 60
        echo "Changing fan speed"
        /opt/tms/bin/mdreq action /system/chassis/actions/set-fan-speed fan_module string "/FAN/FAN" fan_number int8 1 fan_speed int8 27 set_max uint8 50
        /opt/tms/bin/mdreq action /system/chassis/actions/set-fan-speed fan_module string "/FAN/FAN" fan_number int8 2 fan_speed int8 27 set_max uint8 50
        /opt/tms/bin/mdreq action /system/chassis/actions/set-fan-speed fan_module string "/FAN/FAN" fan_number int8 3 fan_speed int8 27 set_max uint8 50
        /opt/tms/bin/mdreq action /system/chassis/actions/set-fan-speed fan_module string "/FAN/FAN" fan_number int8 4 fan_speed int8 27 set_max uint8 50
        /opt/tms/bin/mdreq action /system/chassis/actions/set-fan-speed fan_module string "/PS1/FAN" fan_number int8 1 fan_speed int8 27 set_max uint8 50
        /opt/tms/bin/mdreq action /system/chassis/actions/set-fan-speed fan_module string "/PS2/FAN" fan_number int8 1 fan_speed int8 27 set_max uint8 50
        break
     fi
done

Hi there. I did successfully change the fan speed but when I tried to get the fan speed less than 27, the switch become unresponsive running full speed the fans for a few seconds and then turning them off again. It is stuck in a loop and it doesn't let me edit the fan speed. Any ideas of how to access a cli to bring the fan speed back to 27?

 

[NablaSquaredG]

Which need be converted? all supported by MLX-OS?

SX6036,SX6018,SX6012 can come as EMC versions (need to be converted) or normal Mellanox (no conversion necessary). haven't seen SX6720 and SX6710 as OEMed versions yet.

 

[jerry_zzh]

Hello all,

I used serial login MSB7800 because i don not used eth.I found the following situation.
I didn't find the MAC address of mgmt0 and mgmt1.
Is there a way to solve it? If there is no way to solve it, he may only become MSB7890.

[ OK ] Reached target Login Prompts.
[ 18.685541] tc_ingress_policy[3056]: Cannot find device "mgmts0"
[ 18.696435] sh[3072]: Start Chassis HW management service.
[ 18.703259] tc_ingress_policy[3056]: mDNS policing rate=4000kbit burst=400k
[ OK ] Started SYSV: Starts and stops TC ingerss policy.
[ 18.715796] tc_ingress_policy[3056]: Cannot find device "mgmts0"
[ 18.727741] tc_ingress_policy[3056]: Ingress policing enable on interface mgmts0 rate=9000kbit burst=900k
Starting SYSV: Make addiditional co...of second serial port (ttyS1)...
[ 18.739411] tc_ingress_policy[3056]: Cannot find device "mgmts0"

 

[aho069]

Hello everyone,

Is there anyone familiar with the genlicense command?
I am not sure what to enter in the secret.
I am having trouble because the output license does not become valid.

 

[Zar1n]

Hello everybody.

Does anybody have MLNX-OS firmware for SX6720 ? My new switch returns this message in CLI

Reboot and Select proper Boot device
or Insert Boot Media in selected Boot device and press a key

Thank you.

 

[Marc_]

Morning all, Currently running a successfully converted EMC (sx6018) from the guide here. Looking to play around with the licenses and see if certain things in the network could be improved. Looking at the list of availble options for enabling, am I right in thinking that options 48 through 58 can all be enabled on this version of switch? Obviously not the speed limit options (currently running options, 51,53 & 58).

 

[klui]

I finally had time late this week, like last time during COVID, to convert some more SX6012s I have and didn't like the fact that I need to spend 15 minutes copying and verifying the manifest for each ROOT location then blow the second location away for direct upgrade to 8012 and create unnecessary flash wear.

Has anyone played around with the manufacture.sh option -O LOCATION? The usage states it skips that location. Does it allow the entire partition layout to be created but copy/verify files? Or not even create the partition? I looked at it late last night and didn't spend much time analyzing it and its supporting scripts (writeimage, layout_settings, customer_rootflop). There are also a couple of other layouts like "shared config," "minimal" that one could choose like one root partition.

There were members who stated they waited a long time at the first Modules are being configured after manufacture. Thankfully I didn't have that problem. At most my switches took less than ~10 minutes, even for one that had a dozen (exactly) bad blocks while the flash was being formatted. I'm sharing my times for your reference. I used @Stephan's Cliff-Notes version but his suggestion to use dhcp while in U-Boot didn't work for me. It was looking for BOOTP and I didn't place the kernel, rootfs, and dft files in my tftproot root on a VM. I used @dodgy route's suggestion of using a static IP.

Description	Start	End
Manufacture	7:22	8:13
Prepare Flash​	~7:22	7:37
copy ROOT_1​	7:37	7:49
copy ROOT_2​	7:49	8:01
copy VAR​	8:01	8:03
Image verify​	8:04	8:09 (location 1) 8:13 (location 2)
First boot	8:18
Modules configured		8:27
Upgrade extract image	8:49	9:03

I have 1 more I haven't done, which I used the non-manufacture procedure so if there's anyone who's played around with it please share. Otherwise I'll try adding -O ROOT_2 -O BOOT_2 as additional arguments after reattempting to analyze this weekend.

== Extracting for location ROOT_1 onto /dev/mtdblock6
== Extracting for location ROOT_2 onto /dev/mtdblock7
== Extracting for location CONFIG_1 onto /dev/mtdblock8
== Extracting for location VAR_1 onto /dev/mtdblock9
== Extracting for location BOOT_1 onto /dev/mtdblock6
== Extracting for location BOOT_2 onto /dev/mtdblock7

 

[klui]

Well what do you know, X-mas in February.

A very generous person sent me the correct PowerPC4xx BDI 2000 firmware for this platform. Even the very last release 1.25 from 2014. I am so thrilled. Like an archeologist who just discovered a new entrance to a pyramid. For posterity, reading this on archive.org around the year 2030, the files you need are b20pp4gd.125 (firmware) and pp4jed20.104 or pp4jed21.104 (CPLD) depending whether you have rev B oder rev C BDI 2000.

So I bought a BDI 2000 rev C from Texas on ebay yesterday. Just like the SX6012 itself a device built to last. Rev C will work down to 1.8V system Vcc instead of just 3.0V with Rev B. Taken together with some 13 QSFP nickel plugs by TE to complete my collection, this ran the same amount of what I paid for the switches. But just 10 years earlier it would have cost 20k for two switches with licenses and another (guessing) 10k for the BDI. Two pennies on the dollar.

Will write a BDI config for it and should I get it working, will post it here.

I've also taken a closer look at the 16-pin "CPU BDI" socket yesterday and at least GND and Vcc_target (3.3V) appear to be as specified in the BDI documentation. They talk about a "PowerPC 4xx JTAG connector specification" and by the looks of it in other ppc405 PDFs I found, this appears to have been proposed by IBM for their RISCWatch debugger.

Hey @Stephan, would you mind providing some tips on how to reflash the NAND for the 6012? Specifically the files you referenced above at archive.org and how to connect a BDI 2000/C to the 6012. Guess I'll need to buy one now.

Haha, I was a little too enthusiastic in trying to convert straight to 3.4.8012 and ran into the "out of space" error. Actually it was unexpected EOF from tar. Should have reissued the manufacture command again. I just thought about rebooting into UBOOT and change the RAMDISK size... I rebooted and realized too late the manufacture process wipes all MTD partitions and starts writing beginning with ROOT_1, KERNEL_1, ... instead of UBOOT and UBOOTVAR. I'm sure my subconscious knew because I scanned the logs for my other conversions but it didn't tell me as I was awake.

D'oh! So my remaining 6012 doesn't have any console output because UBOOT and UBOOTVAR are empty!

Thanks!

 

[Stephan]

No problem, check DM. The rescue bundle includes a README. Let me know if you live within EU and want to borrow my BDI 2000.

Edit: There is nothing relevant on archive.org for this. Also U-Boot is in NOR flash. Probably best to just cross flash and not touch partitioning ever again. Developers at Mellanox for PPC have retired, died, moved over to NVidia, or went to work elsewhere. Highly unlikely there will ever be an update newer than 3.4.8012.

 

[klui]

Thanks @Stephan. I'm across the pond from you but very much appreciate your offer. Will update here.

As for -O LOCATION, the script doesn't format the MDTMTD partition.

EDIT: I probably purchased from the same person as you in TX!
EDIT: Right you are about NOR. I appreciate the correction.

From kernel boot of 3.4.0012:

Creating 6 MTD partitions on "4ff000000.nor_flash":
0x00000000-0x001e0000 : "KERNEL_1"
0x001e0000-0x00200000 : "FDT_1"
0x00200000-0x003e0000 : "KERNEL_2"
0x003e0000-0x00400000 : "FDT_2"
0x00f80000-0x00fa0000 : "UBOOTENV"
0x00fa0000-0x01000000 : "UBOOT"
NAND device: Manufacturer ID: 0xec, Chip ID: 0xd3 (Samsung NAND 1GiB 3,3V 8-bit)
2 NAND chips detected
Scanning device for bad blocks
Bad eraseblock 36 at 0x00480000
Bad eraseblock 1564 at 0x0c380000
Bad eraseblock 6022 at 0x2f0c0000
Bad eraseblock 8292 at 0x40c80000
Bad eraseblock 8866 at 0x45440000
Bad eraseblock 8872 at 0x45500000
Bad eraseblock 8999 at 0x464e0000
Bad eraseblock 10272 at 0x50400000
Bad eraseblock 11970 at 0x5d840000
Bad eraseblock 12733 at 0x637a0000
Bad eraseblock 14543 at 0x719e0000
Bad eraseblock 14885 at 0x744a0000
Bad eraseblock 15942 at 0x7c8c0000
Creating 4 MTD partitions on "4e0000000.ndfc.nand":
0x00000000-0x20000000 : "ROOT_1"
0x20000000-0x40000000 : "ROOT_2"
0x40000000-0x46400000 : "CONFIG"
0x46400000-0x7c000000 : "VAR"

 

[NablaSquaredG]

In case anyone wants to mod SX6036T (Limited to 40G) to full 56G:

Steps:
1. Flash correct Switch Firmware (For me it was mfa_extract.py fw-SX-rel-9_4_5110-FIT.mfa MT_1530310031, MT_1530310031 is MSX6036G-xxxS_Ax SwitchX-2 based 36-port QSFP 56GbE Managed InfiniBand to Ethernet gateway system)

2. Mod System FRU EEPROM (located at 0x51)
Starting at byte 0x13E, you will see the bytes 03 03 01 01 (corresponding to FEATURE_EN_6 to FEATURE_EN_9)
Those bytes need to be changed to 04 04 02 02

Oh and you might want to change the PSID to MSX6036G-2SFS instead of MSX6036T-1SFS or so.

Flash the modded FRU EEPROM and you're good to go.

 

[NablaSquaredG]

On this note: Does anyone have genuine SX6036G and could dump the FRU EEPROM? I'd like to see what the SX6036G has a FEATURE_EN_X

 

[klui]

Thanks @Stephan. I'm across the pond from you but very much appreciate your offer. Will update here.

Recovered and converted.

 

[klui]

The manufacture of 3.6.8012 failing was bugging me so I spent some time on it.

The RAMdisk size 262144 suggested by members should be fine but my tests used 409600. The problem is the tmpfs /sbin/writeimage.sh creates is too small. The version of gunzip and zcat fails with error message but return code is 0 when it writes the uncompressed tarball to tmpfs.

==== Uncompressing source image file: /tmp/mnt_image_wi/tmpfs/unzip/image-PPC_M460EX-ppc-m460ex-20190222-075342.tgz to /tmp/mnt_image_wi/tmpfs/unzip/image-PPC_M460EX-3.6.8012.tar
zcat: write: No space left on device
zcat: crc error
zcat: Incorrect length

# cat /tmp/mnt_image_wi/tmpfs/unzip/image-PPC_M460EX-ppc-m460ex-20190222-075342.tgz | zcat - > /tmp/mnt_image_wi/tmpfs/unzip/image-PPC_M460EX-ppc-m460ex-20190222-075342.tar || export FAILURE=1
zcat: write: No space left on device
zcat: crc error
zcat: Incorrect length
# echo $?
0
# echo $FAILURE
0
# rm /tmp/mnt_image_wi/tmpfs/unzip/image-PPC_M460EX-ppc-m460ex-20190222-075342.tar
# zcat /tmp/mnt_image_wi/tmpfs/unzip/image-PPC_M460EX-ppc-m460ex-20190222-075342.tgz > /tmp/mnt_image_wi/tmpfs/unzip/image-PPC_M460EX-ppc-m460ex-20190222-075342.tar || export FAILURE=1
zcat: write: No space left on device
zcat: crc error
zcat: Incorrect length
# echo $?
0
# echo $FAILURE
0
# rm /tmp/mnt_image_wi/tmpfs/unzip/image-PPC_M460EX-ppc-m460ex-20190222-075342.tar
# gunzip /tmp/mnt_image_wi/tmpfs/unzip/image-PPC_M460EX-ppc-m460ex-20190222-075342.tgz || export FAILURE=1
gunzip: write: No space left on device
gunzip: crc error
gunzip: Incorrect length
# echo $?
0
# echo $FAILURE
0

For some reason even though there are only a handful of places where that environment variable is set, changing it to a larger value didn't work. I thought it was using the value in /etc/customer_rootflop.sh. In the end I just hardcoded the value in /sbin/writeimage.sh. 3.6.8012's manufacture/writeimage appears to have the same limitation using 3.4.0012's distribution. The last version has additional definitions for X86 and Onyx images. I didn't use 8012's distro (fdt, kernel, rootfs) files.

# sed 's-/sbin/writeimage-/tmp/writeimage-' /sbin/manufacture.sh > /tmp/manufacture.sh
# sed 's-${TMPFS_SIZE_MB}M,mode=700-1500M,mode=700-' /sbin/writeimage.sh > /tmp/writeimage.sh

# grep writeimage.sh /tmp/manufacture.sh
WRITEIMAGE=/tmp/writeimage.sh
# grep 'M,mode=700' /tmp/writeimage.sh
   mount -t tmpfs -o size=1500M,mode=700 none ${target_dir} || FAILURE=1

# /tmp/manufacture.sh .... http://..../image-PPC_M460EX-3.6.8012.img

==== Verifying image integrity for image-PPC_M460EX-3.6.8012.img
gpg: Signature made Fri Feb 22 08:17:41 2019 UTC using RSA key ID E37FEDA2
gpg: Good signature from "support <support@mellanox.com>"
- Unpacked directory is /tmp/mnt_image_wi/tmpfs/unzip
- Verifying hashes
gpg: Signature made Fri Feb 22 08:17:41 2019 UTC using RSA key ID E37FEDA2
gpg: Good signature from "support <support@mellanox.com>"
- Done
== Running version: SX_PPC_M460EX SX_3.2.0100 2012-03-06 22:23:53 ppc
== Image version:   PPC_M460EX 3.6.8012 2019-02-22 07:53:42 ppc
== Image size: 357 MB / 1115 MB uncompressed
==== Uncompressing source image file: /tmp/mnt_image_wi/tmpfs/unzip/image-PPC_M460EX-ppc-m460ex-20190222-075342.tgz to /tmp/mnt_image_wi/tmpfs/unzip/image-PPC_M460EX-3.6.8012.tar
==== Disk partitioning

 

[NablaSquaredG]

The manufacture of 3.6.8012 failing was bugging me so I spent some time on it.

I gave up at some point and used the

-t: don't use tmpfs for a working area during manufacture  will attempt to use space in /var

flag. It just takes forever then…

 

[klui]

Yeah, I didn't want to wear out the NAND more than necessary although it's using the part allocated for /var.

It's nice -B could be left out so U-Boot can't be clobbered after conversion.

 

[NablaSquaredG]

Yeah, I didn't want to wear out the NAND more than necessary although it's using the part allocated for /var.

Wasn't the NAND MLC or something? I wouldn't be worried about destroying MLC NAND. Also, you don't do that procedure too often. Usually just when you're the last owner and upgrade to 3.6.8012 and then never again