Beware of EMC switches sold as Mellanox SX6XXX on eBay

[JTF195]

I have no idea what his guide looks like nowadays, its been a while
But the one I followed basically worked with a few hiccups (that should be rectified now).

Which post contains your current state?

After getting past the tty boot loop with the trick above, I'm sitting at 'Step 3: Creating the file systems on the switch and extracting the distro'

I immediately ran into an issue. The commands

mount -t proc none /proc
mount -t sysfs none /sys

both fail.

I'm not really sure why, and I also noticed they were already included in the /etc/init.d/rcS file in the jffs2 image, so it's possible they've already been run during init?

I've been taking a break from working on this for a while.

I believe my problem is flashed it with the HP firmware not the mellanox firmware. (My fault I was googling for mlx 3.6.1002 and found a copy on hp's support site but its for their rebrands I believe)

I'm thinking this might be part of my issue as well. I grabbed the one from the HP support site as well, but mpogr confirmed it was the correct image.

What differences have you noticed? (And would you be willing to send me a copy by chance?)

 

[Terry Wallace]

JTF,
Those first two commands failed because they are already mounted which you can check by running 'mount' , you can safely ignore those.
The only reason I suspect I got the wrong firmware was becuase when I burned it it had a HPxxxxxxxx SID rather than the MLXxxxxxx SID and now the MLX-OS on the switch says there is no asic switch found.

I'll PM you about the image

 

[Rand__]

Sent this in a PM but realized I should have put it in a thread reply.

I'm actually following a guide from @mpogr. He sent it to me and the files. The guide I'm following has no "Step 8" how to locally flash the firmware from the switch itself.

Step 8: (optional) Flash Firmware locally

To flash the firmware locally you need shell access to the running MLNX-OS

The steps are

-Boot into scratch linux and adjust admin shell in passwd to /bin/bash (you will need to remove the symlink of /mnt/root2/etc/passwd and create a local copy in /mnt/root2/etc).

-transfer created firmware file to the switch using tftp or scp (either in scratch Linux or MLNX-OS)

-Boot MLNX-OS

-identify target device

mst status

-make sure one more time you have the correct image

flint –I <firmwarename>.bin q

-flash firmware

flint –allow_psid_change –override_cache_replacement –d <device> -I <firmwarename>.bin  b

Reboot the switch.

 

[Terry Wallace]

will try it now.. thanks

 

[Rand__]

welcome. Its actually a part I added (with a few other comments/improvements) which I thought would make their way into the "official" guide but might not have. GL

 

[Terry Wallace]

And we're up and running.. Thanks to @mpogr and @rand_ for the help.

Has anyone tried upgrading the firmware the normal way using the normal mallanox OS upgrades after getting it converted ?
say going from 3.6.1002 upto 3.6.5009 ? Or will I run into trouble ? Just thought I would inquire.

Thanks again to all

Terry

 

[Rand__]

Congrats.
I have not tried since I expect trouble.
Anything juicy in the newer fw?

 

[Kal G]

Congrats.
I have not tried since I expect trouble.
Anything juicy in the newer fw?

Bare metal VXLAN is available starting in version 3.6.3004. Improvements to VXLAN were made in 3.6.4006.

 

[jspuij]

And another succesful conversion. Many thanks to mpogr for providing the neccessary tools!
Stupid question maybe, I can't seem to get the link speed on infiniband from 40Gb to 56 Gb/s. Switch seems to support 56Gb/s according to it's profile, but cannot change port speed. Anybody a tip?

Edit: nevermind, then ibportstate <lid> <port> espeed 1 command and ibportstate <lid> <port> reset did the trick. However it takes a minute or so to reconfigure the link in my case, and by that time I had disabled and enabled it already and was back to 40Gbit.

 

[jspuij]

I'm trying to get two switches to form a HA IB cluster. Joining the second node fails however with the following messages in the log:

Nov  2 15:18:10 switch1 clusterd[25872]: [clusterd.ERR]: HMAC verification failed
Nov  2 15:18:10 switch1 clusterd[25872]: [clusterd.ERR]: cl_verify_hmac_md5(), cl_comm.c:1503, build 1: Error code 14613 returned
Nov  2 15:18:10 switch1 clusterd[25872]: [clusterd.ERR]: cl_ccl_msg_recv(), cl_comm.c:1554, build 1: HMAC verification failed

Could this have anything to do with the shared secret being different between the two switches and does it mean I have to reinstall one of the switches to share the same secret?

Update: indeed the shared secret during database setup has to be the same:

/mfg/mfdb/cluster/config/shared-secret

Note that a running system has this secret copied to the initial database, and can be adjusted by changing the initial database (if you still have access to a shell):

mddbreq -c /config/db/initial set modify - /cluster/config/shared-secret string <YOURSECRETSTRING>

This unfortunately means that converted switches will never be able to HA with a genuine mellanox switch unless somebody is willing to extract the Mellanox shared secret from a genuine switch.

 

[metag]

Just to report another successful conversion. Cannot thank @mpogr enough for the work and detailed guide! I also want to thank @JSLEnterprises for the base image, @Rand__ for the local firmware upgrade instructions, also everyone else sharing their experience in this thread: a lot of collective wisdom there!

 

[cecil1783]

Hey all,

I'm having some troubles flashing my switch. I've gotten everything up through step 6 of the guide, but when I try to boot into the new OS I get an endless stream of errors:

U-Boot 2009.01 SX_PPC_M460EX SX_3.2.0330-82-EMC ppc (Feb 27 2013 - 12:13:42)

CPU:   AMCC PowerPC 460EX Rev. B at 1000 MHz (PLB=166, OPB=83, EBC=83 MHz)
       Security/Kasumi support
       Bootstrap Option H - Boot ROM Location I2C (Addr 0x52)
       Internal PCI arbiter disabled
       32 kB I-Cache 32 kB D-Cache
Board: Mellanox PPC460EX Board
FDEF:  No
I2C:   ready
DRAM:   2 GB (ECC enabled, 333 MHz, CL3)
FLASH: 16 MB
NAND:  1024 MiB
PCI:   Bus Dev VenId DevId Class Int
PCIE0: link is not up.
PCIE1: successfully set as root-complex
        01  00  15b3  c738  0c06  00
Net:   ppc_4xx_eth0, ppc_4xx_eth1
Hit any key to stop autoboot:  0
INIT: version 2.86 booting

Starting: PPC_M460EX 3.6.1002 2016-06-09 20:24:26 ppc
Starting udev: [  OK  ]
Setting clock  (utc): Thu Nov 15 05:08:18 UTC 2018 [  OK  ]
Setting hostname localhost:  [  OK  ]
Checking filesystems
Checking all file systems.
[  OK  ]
Remounting root filesystem in read-write mode:  [  OK  ]
Mounting local filesystems:  [  OK  ]
Running vpart script:  [  OK  ]
Applying file system skeletons: base_var base_config .
Running firstboot script usage: /sbin/aiset.sh -i [-l NEXT_BOOT_ID] [-p MD5_PASSWORD] [-r] [-f {true,false}] [-F] [-E]
usage: /sbin/aiset.sh -m -d BOOT_DISK [-L LAYOUT] [-l NEXT_BOOT_ID]
          [-p MD5_PASSWORD]

-i: not running at manufacture time (generally image install)
-m: running at manufacture time

-l NEXT_BOOT_ID: image location to boot from: 1 or 2
-d BOOT_DISK: (mfg only) /dev/sda or /dev/hda
-L LAYOUT: (mfg only) image layout, like STD
-w HWNAME: (mfg only) hardware name (usually optional on x86)
-p MD5_PASSWORD: MD5 encrypted password
-r: (install only) re-install the bootmgr itself (GRUB or u-boot)
-f {true,false}: enable or disable fallback reboot behavior for next boot
-I IMAGE_LOCATION_ID -s IMAGE_LOCATION_STATE : exclusive with -l
        States are: 0=invalid; 1=active; 2=fallback; 3=manual
-F FIPS: Use this flag to add fips=1 flag for command line run
-E FIPS_DISABLE: User this flag to set fips=0 for command line run

Writes a grub.conf which use the selected next boot location,
   and which contains the installed image version strings.

Generating SSH1 RSA host key: [  OK  ]
Generating SSH2 RSA host key: [  OK  ]
Generating SSH2 DSA host key: [  OK  ]
    Starting sx_low_level_if:
Loading i2c_mux_pca954x  - Success
Loading sx_glue_if  - Success
Loading watchdog  - Success
Loading cpld_handler  - Success
Loading mellaggra_mod  - Success
Loading switchx  - Success
Reloading udev:
Loading SX driver:[  OK  ]
Stopping iss-nvram-mac
Stopping sx_low_level_if
switchx module unloaded
mellaggra_mod module unloaded
cpld_handler module unloaded
watchdog module unloaded
i2c_mux_pca954x module unloaded
mlx system_profile: 3
mlx system_type: SX6018
mlx system_oid: 1.3.6.1.4.1.33049.1.1.1.6018
    Starting sx_low_level_if:
Loading i2c_mux_pca954x  - Success
NOTE: sx_glue_if module is already loaded
Loading watchdog  - Success
Loading cpld_handler  - Success
Loading mellaggra_mod  - Success
Loading switchx  - Success
[  OK  ]
Enabling /etc/fstab swaps:  [  OK  ]
INIT: Entering runlevel: 3
Starting system services
Starting sx_low_level_if:      Starting sx_low_level_if:
NOTE: i2c_mux_pca954x module is already loaded
NOTE: sx_glue_if module is already loaded
NOTE: watchdog module is already loaded
NOTE: cpld_handler module is already loaded
NOTE: mellaggra_mod module is already loaded
NOTE: switchx module is already loaded
[  OK  ]
Starting openibd:  IPoIB configuration for embedded system
Loading SX driver:[  OK  ]
Loading HCA driver and Access Layer:[  OK  ]
Setting up InfiniBand network interfaces:
Setting up service network . . .[  done  ]
Reloading udev:
[  OK  ]
Starting system logger: [  OK  ]
Starting kernel logger: [  OK  ]
Starting fips_post:  [  OK  ]
Renaming: no changes required
/etc/rc3.d/S15rename_ifs: line 326: return: can only `return' from a function or sourced script
Renaming: no changes for: MAC: 00:02:C9:64:18:4C ifindex: 2 name: mgmt0
Renaming: no changes for: MAC: 00:02:C9:64:18:4D ifindex: 3 name: mgmt1
Running renaming interfaces
/etc/rc3.d/S15rename_ifs: line 384: do_rename_ifs: comma__nand_correct_data: uncorrectable ECC errornd not found
Checking for une
__nand_correct_data: uncorrectable ECC errorxp__nand_correct_data: uncorrectable ECC errorec
__nand_correct_data: uncorrectable ECC errorted shutd__nand_correct_data: uncorrectable ECC errorow
__nand_correct_data: uncorrectable ECC errorn

__nand_correct_data: uncorrectable ECC errorblk_update_request: 22 callbacks suppressed
end_request: I/O error, dev mtdblock6, sector 0
quiet_error: 22 callbacks suppressed
Buffer I/O error on device mtdblock6, logical block 0

Probing for HRNG module
Starting rngd: [  OK  ]
Running syst__nand_correct_data: uncorrectable ECC errorem image: PPC_M460EX 3.6.1002 2016-06-09 20:24:2
__nand_correct_data: uncorrectable ECC error6 ppc
__nand_correct_data: uncorrectable ECC error
__nand_correct_data: uncorrectable ECC error__nand_correct_data: uncorrectable ECC error
__nand_correct_data: uncorrectable ECC error__nand_correct_data: uncorrectable ECC error
__nand_correct_data: uncorrectable ECC errorend_request: I/O error, dev mtdblock6, sector 8
Buffer I/O error on device mtdblock6, logical block 1
__nand_correct_data: uncorrectable ECC error
__nand_correct_data: uncorrectable ECC error__nand_correct_data: uncorrectable ECC error
__nand_correct_data: uncorrectable ECC error__nand_correct_data: uncorrectable ECC error
__nand_correct_data: uncorrectable ECC error__nand_correct_data: uncorrectable ECC error
end_request: I/O error, dev mtdblock6, sector 16
Buffer I/O error on device mtdblock6, logical block 2
__nand_correct_data: uncorrectable ECC error
__nand_correct_data: uncorrectable ECC error__nand_correct_data: uncorrectable ECC error
__nand_correct_data: uncorrectable ECC error__nand_correct_data: uncorrectable ECC error
__nand_correct_data: uncorrectable ECC errorend_request: I/O error, dev mtdblock6, sector 24
Buffer I/O error on device mtdblock6, logical block 3
Applying initial configuration: __nand_correct_data: uncorrectable ECC error
__nand_correct_data: uncorrectable ECC error__nand_correct_data: uncorrectable ECC error
__nand_correct_data: uncorrectable ECC error__nand_correct_data: uncorrectable ECC error
__nand_correct_data: uncorrectable ECC error__nand_correct_data: uncorrectable ECC error
__nand_correct_data: uncorrectable ECC errorend_request: I/O error, dev mtdblock6, sector 0
Buffer I/O error on device mtdblock6, logical block 0
__nand_correct_data: uncorrectable ECC error
__nand_correct_data: uncorrectable ECC error__nand_correct_data: uncorrectable ECC error
__nand_correct_data: uncorrectable ECC error__nand_correct_data: uncorrectable ECC error
__nand_correct_data: uncorrectable ECC error__nand_correct_data: uncorrectable ECC error
__nand_correct_data: uncorrectable ECC errorend_request: I/O error, dev mtdblock6, sector 0

I can still boot into the EMC OS and the scratch Linux area.

Any thoughts? Did I edit a file wrong somewhere?


Thanks!

 

[metag]

Hey all,

I'm having some troubles flashing my switch. I've gotten everything up through step 6 of the guide, but when I try to boot into the new OS I get an endless stream of errors:

I can still boot into the EMC OS and the scratch Linux area.

Any thoughts? Did I edit a file wrong somewhere?


Thanks!

It's normal. I just waited it out. I believe it's a first time thing.

 

[cecil1783]

Well I'll be a monkey's uncle...that worked. Thanks a ton for the quick reply!

 

[Poco]

Anyone interested in conversion, PM me.

Hi.

I just bought one of these SX6005 switches before seeing this post..
If you're still able to help with this I'd greatly appreciate it.

Kind regards.

 

[cecil1783]

Hi.

I just bought one of these SX6005 switches before seeing this post..
If you're still able to help with this I'd greatly appreciate it.

Kind regards.

The SX6005 cannot be converted to a managed switch, as it does not have the hardware necessary. You should be able to flash it to a normal SX6005 firmware, though, using the publicly available firmware (http://www.mellanox.com/page/firmware_table_SwitchX) and flint (http://www.mellanox.com/page/management_tools). You will need to have opensm running on an infiniband connected machine to do the flash.

 

[herby]

The SX6005 cannot be converted to a managed switch, as it does not have the hardware necessary. You should be able to flash it to a normal SX6005 firmware, though, using the publicly available firmware (http://www.mellanox.com/page/firmware_table_SwitchX) and flint (http://www.mellanox.com/page/management_tools). You will need to have opensm running on an infiniband connected machine to do the flash.

Would this process be less challenging than the flashing a SX6012/6018/6036?

For my application 40GbE unmanaged would be fine and the SX6005 is cheaper.

 

[cecil1783]

Would this process be less challenging than the flashing a SX6012/6018/6036?

For my application 40GbE unmanaged would be fine and the SX6005 is cheaper.

The unmanaged switches (SX6xx5) are not capable of ethernet. Ethernet mode requires a license on the managed switches (though no license is actually required on our flashed switches).

To your first question, yes, flashing the unmanaged switches would be considerably easier.

edit: another thought, if you've got ConnectX VPI cards at all ends, and you're willing to run opensm on one system somewhere, you could run IPoIB on the unmanaged switches. That nets the same result, but requires everything throughout support it.

aa

 

[herby]

The unmanaged switches (SX6xx5) are not capable of ethernet. Ethernet mode requires a license on the managed switches (though no license is actually required on our flashed switches).

To your first question, yes, flashing the unmanaged switches would be considerably easier.

edit: another thought, if you've got ConnectX VPI cards at all ends, and you're willing to run opensm on one system somewhere, you could running IPoIB on the unmanaged switches. That nets the same result, but requires everything throughout support it.

aa

Oh well, that's too bad. I'm running FreeNAS at the moment and it doesn't do InfiniBand.
Thanks for the info, I'll have to think about if I want to gamble on one of the SX6012's.

 

[cecil1783]

I just finished setting up my FreeNAS, actually, and I was pretty disappointed to see there was no IB support. It does handle 56GbE pretty well (everything connected, anyway, haven't had a chance to test throughput yet), though.


aa