Beware of EMC switches sold as Mellanox SX6XXX on eBay

t2_tony

New Member
Mar 29, 2020
4
0
1
Los Angeles Metro
Need some guidance. Ran into a hiccup at Step 6.2.B.

[UPDATE: Just re-read info on 10th page of this thread...got it] w.r.t.: ECC error terminal spew

I have initial success using 8012: used the alternate chroot approach for step 6.2B
then dropped into /dev/mtdblock6 to dwiddle

[UPDATE-2] Moving the ball - true to the guide:
 

Attachments

Last edited:

Eong

Member
Dec 24, 2019
35
7
8
I had a look at the files and I saw that ibd file is missing. We don't need to replace it or we can use the old one? I guess there is an update between 1002 and 8012 versions of the ibd.

I think the best way of conversion would be to dump the flash from Mellanox 6012 or maybe it's working from a 6018 or 6036 and try to flash directly those. After that update the FRU part so it can be seen as 6012. Or maybe the management board can identify the chassis type automatically at first reboot.

I tried for a long time to find a Mellanox 6012 model but they are rare and costs a lot compared to the EMC models.
You do not need to replace ibt. I didn’t see any issue for now.
It does have a limitation for the LR4, I can confirm that. It’s complicated. Those compatible ones may work or not. The one from FS doesn’t work. I can not try all of the brands. Too expensive. :(
FRU information from 6036 may help. I am still waiting for mine to arrive.
 

devin_mm

New Member
May 7, 2019
19
1
3
I need to fix the firmware on my SX6018, it looks like the previous owner messed around with the firmware and now it's not loading properly.
The issues I am running into are:
the web interface is showing the switch as an SX6036
the LEDs on the front panel show all green even if I only have one PSU installed or if I unplug the fan module
show voltage just shows the message "No managed power meters found."
show temperature comes up empty
Is there a way to fix this?
 

Eong

Member
Dec 24, 2019
35
7
8
I need to fix the firmware on my SX6018, it looks like the previous owner messed around with the firmware and now it's not loading properly.
The issues I am running into are:
the web interface is showing the switch as an SX6036
the LEDs on the front panel show all green even if I only have one PSU installed or if I unplug the fan module
show voltage just shows the message "No managed power meters found."
show temperature comes up empty
Is there a way to fix this?
I think it's not the firmware, it is the modified binaries. You may have to modify them yourself. Or you can send me a SX6018. :)
 
Apr 21, 2016
47
17
8
40
Hi all,

what is the easiest (and cheapest) way to connect one of these to an existing gigabit switch?

or is there breakout cables to rj45? (i tried to find some but couldn't)
This is obviously to keep the rest of the network running if the ethernet switches have no SFP ports.

Thanks
I can confirm that a breakout dac cable (mine is cisco branded) works - 1/10g possible. I haven't tried a media converter (sfp->rj45) yet.
 

Eong

Member
Dec 24, 2019
35
7
8
I can confirm that a breakout dac cable (mine is cisco branded) works - 1/10g possible. I haven't tried a media converter (sfp->rj45) yet.
You need the MAM1Q00A-QSA and a ethernet SFP module. I tried different brands and all of them work with MAM1Q00A-QSA.
I bought 20 MAM1Q00A-QSA when I found really a nice price.
 

devin_mm

New Member
May 7, 2019
19
1
3
I think it's not the firmware, it is the modified binaries. You may have to modify them yourself. Or you can send me a SX6018. :)
Haha, I assume I can figure out what I need to do to get/modify the binaries in this thread. I'll have to work my way through it.
 

Labs

Member
Mar 21, 2019
46
7
8
You do not need to replace ibt. I didn’t see any issue for now.
It does have a limitation for the LR4, I can confirm that. It’s complicated. Those compatible ones may work or not. The one from FS doesn’t work. I can not try all of the brands. Too expensive. :(
FRU information from 6036 may help. I am still waiting for mine to arrive.
I think the limitation for LR4 is that it doesn't allow non-Mellanox QSFPs to work. Probably it's working fine with Mellanox original ones.

I will try to power on my 6036 and send you the FRU info. From what I remember mine was HP branded but it was running the Mellanox OS standard version.
 

Eong

Member
Dec 24, 2019
35
7
8
I think the limitation for LR4 is that it doesn't allow non-Mellanox QSFPs to work. Probably it's working fine with Mellanox original ones.

I will try to power on my 6036 and send you the FRU info. From what I remember mine was HP branded but it was running the Mellanox OS standard version.
Looking forward to it!
 

t2_tony

New Member
Mar 29, 2020
4
0
1
Los Angeles Metro
Really need some advice now. here it is


# sho boot

Installed images:
Partition 1:
version: (Could not determine image version -- install in progress?)

Partition 2:
version: PPC_M460EX 3.6.8012 2019-02-22 07:53:42 ppc

Last boot partition : 2
Next boot partition : 1
Serve image files via HTTP/HTTPS: no

Boot manager password is set.

Image signing : trusted signature always required
Admin require signed images: yes

Settings for next boot only:
Fallback reboot on configuration failure: yes (default)


# show system capabilities
((goose egg))


Shot over to grab the free 40/56 lic to test, and the site down for maintenance. the ir subdomain is already hitting nVidia landing page
so taking the rest of the night off

pm me if u post, i recognize a lot of activity with the fans.

Thanks in advance.
 

sbhsu

New Member
Mar 3, 2016
4
0
1
38
I upgrade 3.6.1002 to 3.6.8012,and got follow error on first boot.

INIT: version 2.86 booting
mount: /lib/libblkid.so.1: no version information available (required by /lib/libmount.so.1)
mount: /lib/libblkid.so.1: no version information available (required by /lib/libmount.so.1)
mount: /lib/libblkid.so.1: no version information available (required by /lib/libmount.so.1)

Starting: PPC_M460EX 3.6.8012 2019-02-22 07:53:42 ppc
Starting udev: mount: /lib/libblkid.so.1: no version information available (required by /lib/libmount.so.1)
mount: /lib/libblkid.so.1: no version information available (required by /lib/libmount.so.1)
mount: /lib/libblkid.so.1: no version information available (required by /lib/libmount.so.1)
error initializing udevd socket
Wait timeout. Will continue in the background.[FAILED]
Setting hostname localhost: [ OK ]
Checking filesystems
fsck: /lib/libblkid.so.1: no version information available (required by fsck)
fsck: /lib/libblkid.so.1: no version information available (required by fsck)
fsck: /lib/libblkid.so.1: no version information available (required by /lib/libmount.so.1)
fsck: /lib/libblkid.so.1: no version information available (required by /lib/libmount.so.1)
fsck: /lib/libblkid.so.1: no version information available (required by /lib/libmount.so.1)
/dev/mtdblock7: nonexistent device ("nofail" fstab option may be used to skip this device)
/dev/mtdblock8: nonexistent device ("nofail" fstab option may be used to skip this device)
/dev/mtdblock9: nonexistent device ("nofail" fstab option may be used to skip this device)
Checking all file systems.
[ OK ]
Remounting root filesystem in read-write mode: mount: /lib/libblkid.so.1: no version information available (required by /lib/libmount.so.1)
mount: /lib/libblkid.so.1: no version information available (required by /lib/libmount.so.1)
mount: /lib/libblkid.so.1: no version information available (required by /lib/libmount.so.1)
[ OK ]
mount: /lib/libblkid.so.1: no version information available (required by /lib/libmount.so.1)
mount: /lib/libblkid.so.1: no version information available (required by /lib/libmount.so.1)
mount: /lib/libblkid.so.1: no version information available (required by /lib/libmount.so.1)
Mounting local filesystems: mount: /lib/libblkid.so.1: no version information available (required by /lib/libmount.so.1)
mount: /lib/libblkid.so.1: no version information available (required by /lib/libmount.so.1)
mount: /lib/libblkid.so.1: no version information available (required by /lib/libmount.so.1)
mount: special device /dev/mtdblock8 does not exist
mount: special device /dev/mtdblock9 does not exist
[FAILED]
Running vpart script: [ OK ]
Applying file system skeletons: base_var base_config .
Running firstboot script ... uboot and bootsrap updates are required
Re-install updated uboot
-E- could not open device
mount: special device /dev/mtdblock6 does not exist
umount: /tmp/mnt_image_ais/ROOT_1: not mounted
*** Could not zero non-fs MTD partition /dev/mtd4
Rebooting system after uboot upgrade
-E- could not open device
-E- could not open device
usage: /sbin/aiset.sh -i [-l NEXT_BOOT_ID] [-p MD5_PASSWORD] [-r] [-f {true,false}] [-F] [-E]
usage: /sbin/aiset.sh -m -d BOOT_DISK [-L LAYOUT] [-l NEXT_BOOT_ID]
[-p MD5_PASSWORD]

-i: not running at manufacture time (generally image install)
-m: running at manufacture time

-l NEXT_BOOT_ID: image location to boot from: 1 or 2
-d BOOT_DISK: (mfg only) /dev/sda or /dev/hda
-L LAYOUT: (mfg only) image layout, like STD
-w HWNAME: (mfg only) hardware name (usually optional on x86)
-p MD5_PASSWORD: MD5 encrypted password
-r: (install only) re-install the bootmgr itself (GRUB or u-boot)
-f {true,false}: enable or disable fallback reboot behavior for next boot
-I IMAGE_LOCATION_ID -s IMAGE_LOCATION_STATE : exclusive with -l
States are: 0=invalid; 1=active; 2=fallback; 3=manual
-F FIPS: Use this flag to add fips=1 flag for command line run
-E FIPS_DISABLE: User this flag to set fips=0 for command line run

Writes a grub.conf which use the selected next boot location,
and which contains the installed image version strings.

error reading information on service arp_responder: No such file or directory
-E- could not open device
Generating SSH1 RSA host key: [FAILED]
Generating SSH2 RSA host key: [FAILED]
Generating SSH2 DSA host key: [FAILED]
Starting sx_low_level_if:
Loading i2c_mux_pca954x - Failure: 1
Loading sx_glue_if - Failure: 1
Loading watchdog - Failure: 1
Loading cpld_handler - Failure: 1
Loading mellaggra_mod - Failure: 1
Loading switchx - Failure: 1
Loading 8021q:[FAILED]
-E- could not open device
Error: mlxi2c failed: cant read system type
[FAILED]
Enabling /etc/fstab swaps: [ OK ]
Remounting root file
Anyone have the solution?
 

Eong

Member
Dec 24, 2019
35
7
8
I upgrade 3.6.1002 to 3.6.8012,and got follow error on first boot.



Anyone have the solution?
Did you upload the kernel and another file? Check the guide, at the beginning part, the kernel and another files should be uploaded. That’s the reason. Sorry I forgot to mention. Should be added into instructions.
 

Eong

Member
Dec 24, 2019
35
7
8
Really need some advice now. here it is


# sho boot

Installed images:
Partition 1:
version: (Could not determine image version -- install in progress?)

Partition 2:
version: PPC_M460EX 3.6.8012 2019-02-22 07:53:42 ppc

Last boot partition : 2
Next boot partition : 1
Serve image files via HTTP/HTTPS: no

Boot manager password is set.

Image signing : trusted signature always required
Admin require signed images: yes

Settings for next boot only:
Fallback reboot on configuration failure: yes (default)


# show system capabilities
((goose egg))


Shot over to grab the free 40/56 lic to test, and the site down for maintenance. the ir subdomain is already hitting nVidia landing page
so taking the rest of the night off

pm me if u post, i recognize a lot of activity with the fans.

Thanks in advance.
So, the question is?
 

sbhsu

New Member
Mar 3, 2016
4
0
1
38
Did you upload the kernel and another file? Check the guide, at the beginning part, the kernel and another files should be uploaded. That’s the reason. Sorry I forgot to mention. Should be added into instructions.
Hi Eong,

When I update the kernel file on uboot,it working fine finally.

Thank you.
 

t2_tony

New Member
Mar 29, 2020
4
0
1
Los Angeles Metro
Sorry Eong, got exhausted. So should the switch not upgraded with some of the core capabilites ?
While i recognize 'workarounds' are going to be required for L2 and GW, expected the basic
IB at 56GbE since its a F model, 648node, not sure what else.

Assuming for the moment i followed the rooting correctly, what is the procedure to break out from enable into bash?
From there, I can start doing some MFG changes/service starts, but i'm very fuzzy how the core licenses got wiped; these were in the db and not persistent across the upgrade?

Is the ppc model strictly jffs2, or has anybody tried ubifs/squash with success...maybe cleans up NAND ECC errs?
I stumbled over this MTD in the customer_rootflop.sh
# 16 MB NOR Flash:
# /dev/mtd0 [nmp] kernel 1 (raw partition, uImage kernel, cp / dd)
# /dev/mtd1 [nmp] kernel fdt 1 (raw partition, DTB, cp / dd)
# /dev/mtd2 [nmp] kernel 2 (raw partition, uImage kernel, cp / dd)
# /dev/mtd3 [nmp] kernel fdt 2 (raw partition, DTB, cp / dd)
# /dev/mtd4 [nmp] u-boot env (raw partition, U-boot env, cp / dd)
# /dev/mtd5 [nmp] u-boot (raw partition, U-boot, cp / dd)
#
# 1024 MB NAND Flash:
# /dev/mtdblock6 / [root 1] ro root (jffs2)
# /dev/mtdblock7 / [root 2] ro root (jffs2)
# /dev/mtdblock8 /config rw config db (jffs2)
# /dev/mtdblock9 /var rw general, logs (jffs2)

So definitely concerned to see:
|> Partition 1:
|> version: (Could not determine image version -- install in progress?)
is this normal, with this partition ( /dev/mtdblock6 ) ? that first boot into mini-kernel was a 2hr+ terminal flood of ecc errors so i'm second-guessing that i need to touch up a bunch.

I'm new to high-end gear modding for personal use, appreciate any tips & tricks that work.

~t2
 

Eong

Member
Dec 24, 2019
35
7
8
Sorry Eong, got exhausted. So should the switch not upgraded with some of the core capabilites ?
While i recognize 'workarounds' are going to be required for L2 and GW, expected the basic
IB at 56GbE since its a F model, 648node, not sure what else.

Assuming for the moment i followed the rooting correctly, what is the procedure to break out from enable into bash?
From there, I can start doing some MFG changes/service starts, but i'm very fuzzy how the core licenses got wiped; these were in the db and not persistent across the upgrade?

Is the ppc model strictly jffs2, or has anybody tried ubifs/squash with success...maybe cleans up NAND ECC errs?
I stumbled over this MTD in the customer_rootflop.sh
# 16 MB NOR Flash:
# /dev/mtd0 [nmp] kernel 1 (raw partition, uImage kernel, cp / dd)
# /dev/mtd1 [nmp] kernel fdt 1 (raw partition, DTB, cp / dd)
# /dev/mtd2 [nmp] kernel 2 (raw partition, uImage kernel, cp / dd)
# /dev/mtd3 [nmp] kernel fdt 2 (raw partition, DTB, cp / dd)
# /dev/mtd4 [nmp] u-boot env (raw partition, U-boot env, cp / dd)
# /dev/mtd5 [nmp] u-boot (raw partition, U-boot, cp / dd)
#
# 1024 MB NAND Flash:
# /dev/mtdblock6 / [root 1] ro root (jffs2)
# /dev/mtdblock7 / [root 2] ro root (jffs2)
# /dev/mtdblock8 /config rw config db (jffs2)
# /dev/mtdblock9 /var rw general, logs (jffs2)

So definitely concerned to see:
|> Partition 1:
|> version: (Could not determine image version -- install in progress?)
is this normal, with this partition ( /dev/mtdblock6 ) ? that first boot into mini-kernel was a 2hr+ terminal flood of ecc errors so i'm second-guessing that i need to touch up a bunch.

I'm new to high-end gear modding for personal use, appreciate any tips & tricks that work.

~t2
I am not the one created this conversation. I just provided a way to let you upgrade to 1082.
As far as I know, the ecc errors will show up at the first boot, or maybe second. After that, you will not see that much.

You flash your own linux image into mtdblock6, that's normal.

For the bash, just change the passwd file in /etc and give admin /bin/bash other than /opt/xxx/bin/cli. But if you check it by 'ls -al', you will see it is just a link. And the file will be created each time you boot. Cut the connection, that's the 'break'.

Sorry I shoud not discuss anything about license. :)
 
  • Like
Reactions: t2_tony

Freebsd1976

Active Member
Feb 23, 2018
222
34
28
first boot into mini-kernel was a 2hr+ terminal flood of ecc errors so i'm second-guessing that i need to touch up a bunch.

I'm new to high-end gear modding for personal use, appreciate any tips & tricks that work.

~t2
this mean your minios have something wrong ,don't make minios yourself, download instead , read carfully this thread someone one already share his minios image ,download it ,reflash it , and boot to minios only take seconds , not mintues or hours.
 
  • Like
Reactions: t2_tony

Takrbark3

New Member
Dec 17, 2017
18
0
1
39
Hi!

The Mellanox SX6036 is now EOL.
MLNX-OS renamed to ONYX.

Mellanox SBxxxx for infiniband
Mellanox SNxxxx for ethernet

Which Mellanox switch can do "infiniband+ethernet" ?
Which Mellanox switch can do "infiniband+ethernet+FCOE" ?

I could not find clear answers.
 

Rand__

Well-Known Member
Mar 6, 2014
4,585
912
113
None of the new ones can do both unfortunately.

I am sure the SNs can do FCoE ( don't have one, never looked into it though), not sure if the IBs can