Beware of EMC switches sold as Mellanox SX6XXX on eBay

[neggles]

Mine originally came up in VPI mode with the profile set to 3; I found I could switch profiles just fine with the system profile [ib|eth|vpi] config command (I'm running ethernet only)

Can you run show system capabilities and paste the output / see what it says?

 

[Raice]

MSX6012 to 8012 following guide 1.2.

Wow! Did I miss something? Could you point me to this guide?

 

[neggles]

8012 refers to firmware 3.6.8012 - guide/files are here or here

 

[francois69380]

show system capabilities returns nothing.
MSX6012 [standalone: master] (config) # system profile eth-single-switch
Warning - confirming will cause system reboot and all configuration will be deleted
Type 'yes' to confirm profile change: yes
% new profile can't be changed to VPI as it is not enabled by license
- Configuration reset failed
MSX6012 [standalone: master] (config) # show system capabilities
MSX6012 [standalone: master] (config) # exit

 

[klui]

Did your switch originally came with EMC software or is it a real Mellanox MSX6012? Pull out the asset tag below the console port and see if the part number, P/N, contains all digits or does it start with MST?

What does

sh system profile​

output?

 

[francois69380]

Did your switch originally came with EMC software or is it a real Mellanox MSX6012? Pull out the asset tag below the console port and see if the part number, P/N, contains all digits or does it start with MST?

What does

sh system profile​

output?

This was an emc, it was converted ok with mlnx OS 3.6.1002 . I just want to upgrade to firmware 3.6.8012.

MSX6012 [standalone: master] > sh system profile

Profile: vpi-single-switch

 

[klui]

My profile is the same. Just force the ports to ethernet. For example

port 1/1-1/6 type ethernet force​

 

[francois69380]

My profile is the same. Just force the ports to ethernet. For example

port 1/1-1/6 type ethernet force​

I just did a full reinstall and all is OK
Thank you for your help

 

[Grus]

NEED SOME GUIDANCE HERE

Got an EMC SX6012(P/N: 100-886-236-04), want to convert it to an MLNX-OS one.

Followed the Conversion Guide v1.2 from neggles’s website
Finished Step 5 Creating configuration DB nodes,
Then I think I made some mistakes during Step 6.2 Flashing the firmware:
My Linux server had ConnectX-3 VPI card connected to the switch with an FDR cable, I executed

mst status

couldn't find the switchlid from the Linux server – seems that I should use ibnetdiscover but didn’t then.

Again, I wasn't quite sure how to boot into scratch linux (mtdblock6) etc.…

So in a hot head status myself, I skipped the firmware update,

setenv rootdev /dev/mtdblock7
saveenv
run mlxlinux

After that, there were endless

__nand_correct_data: uncorrectable ECC error
...

for at least several hours.

I could get halt somewhere to localhost login: and login:
But admin/admin didn’t work.

Then I tried to redo it from Step 1, after

setenv jffs2_args 'setenv bootargs root=${rootdev} rootfstype=jffs2 rw reset_button=${reset_button} ${image_kernel_args} ${extra_args}'
run mlxlinux

in Step 2, It’s endless

__nand_correct_data: uncorrectable ECC error
end_request: I/O error, dev mtdblock6, sector 0
Buffer I/O error on device mtdblock6, logical block 0
...

I can reset the switch and auto boot into EMC system or get into U-Boot => without password now,
What could I do from here?

ANY HELP IS APPRECIATED!

 

[klui]

I can't help with using IB to flash because I never went that route.

To boot to scratch linux, set your rootdev to /dev/mtdblock6 followed by run mlxlinux.

You should see if you have image_kernel_args set to loglevel=2. This setting will eliminate most of the ECC errors. But you will still get them during first boot from /dev/mtdblock7, but it happens once and subsequent reboots will be fine. During first boot I got a little over 4300 lines of these errors. Just let it spew. But if you don't have loglevel=2 you will get much more.

Your hostname should not be localhost unless you didn't set hostname string.

There's a catch 22 of flashing the firmware. You need to allow /dev/mtdblock7 as the root device to perform first boot so /etc/passwd is set up properly before you can login.

The more efficient way is to

1. Do your conversion - do not remove /mnt/root2/etc/passwd symlink when in scratch linux (booted from /dev/mtdblock6)
2. Ensure loglevel=2 is set, save your U-boot environment
3. Perform first boot of /dev/mtdblock7
   1. The second time sx_low_level_if starts, you need to wait a bit for "Loading switchx" to complete - this happens before the ECC errors are printed
   2. With the EMC firmware, it will take longer to complete after "Modules are being configured" is printed
4. Confirm admin/admin works
5. Reboot and set single user mode booting from /dev/mtdblock7 (https://forums.servethehome.com/ind...-as-mellanox-sx6xxx-on-ebay.10786/post-227694)
6. Flash MLNX-OS firmware
7. Reboot - you can rename /mnt/root2/etc/passwd and copy over /mnt/var/tms/output/passwd if you wish or you can continue using cli as the shell

 

[Grus]

Thanks a lot.

I can't help with using IB to flash because I never went that route.

I met the same problem with using IB to flash FW as Post#337
It seems like not a good idea to try.

To boot to scratch linux, set your rootdev to /dev/mtdblock6 followed by run mlxlinux.

Yeah, I did setenv rootdev /dev/mtdblock7 & saveenv
That's why I got stuck in GUIDEv1.2 Step 2 when trying to redo the conversion
there need a setenv rootdev /dev/mtdblock6 before run mlxlinux

1. Do your conversion - do not remove /mnt/root2/etc/passwd symlink when in scratch linux (booted from /dev/mtdblock6)
2. Ensure loglevel=2 is set, save your U-boot environment
3. Perform first boot of /dev/mtdblock7
   1. The second time sx_low_level_if starts, you need to wait a bit for "Loading switchx" to complete - this happens before the ECC errors are printed
   2. With the EMC firmware, it will take longer to complete after "Modules are being configured" is printed
4. Confirm admin/admin works
5. Reboot and set single user mode booting from /dev/mtdblock7 (https://forums.servethehome.com/ind...-as-mellanox-sx6xxx-on-ebay.10786/post-227694)
6. Flash MLNX-OS firmware
7. Reboot - you can rename /mnt/root2/etc/passwd and copy over /mnt/var/tms/output/passwd if you wish or you can continue using cli as the shell

I managed to Flash MLNX-OS firmware following Post#372
It seems no need to mess with /etc/passwd file.

Now I got the MLNX-OS booted

=> run mlxlinux
INIT: version 2.86 booting

Starting: PPC_M460EX 3.6.8012 2019-02-22 07:53:42 ppc
Starting udev: [ OK ]
Setting clock (utc): Fri Aug 16 09:15:47 UTC 2013 [ OK ]
Setting hostname localhost: [ OK ]
Checking filesystems
Checking all file systems.
[ OK ]
Remounting root filesystem in read-write mode: [ OK ]
Mounting local filesystems: [ OK ]
Running vpart script: [ OK ]
Applying file system skeletons: base_var base_config .
Enabling /etc/fstab swaps: [ OK ]
INIT: Entering runlevel: 3
Starting system services
Starting sx_low_level_if: Starting sx_low_level_if:
Loading i2c_mux_pca954x - Success
Loading sx_glue_if - Success
Loading watchdog - Success
Loading cpld_handler - Success
Loading mellaggra_mod - Success
Loading switchx - Success
[ OK ]
Starting openibd: IPoIB configuration for embedded system
Loading SX driver:[ OK ]
Loading HCA driver and Access Layer:[ OK ]
Setting up InfiniBand network interfaces:
Setting up service network . . .[ done ]
Reloading udev:
[ OK ]
Starting system logger: [ OK ]
Starting kernel logger: [ OK ]
Starting fips_post: [ OK ]
Renaming: no changes required
/etc/rc3.d/S15rename_ifs: line 325: return: can only `return' from a function or sourced script
Renaming: no changes for: MAC: 98:03:9B:AB:3B:8E ifindex: 2 name: mgmt0
Renaming: no changes for: MAC: 98:03:9B:AB:3B:8F ifindex: 3 name: mgmt1
/etc/rc3.d/S15rename_ifs: line 378: syntax error near unexpected token `}'
/etc/rc3.d/S15rename_ifs: line 378: `}'
Checking for unexpected shutdown

Probing for HRNG module
Starting rngd: [ OK ]
Running system image: PPC_M460EX 3.6.8012 2019-02-22 07:53:42 ppc
Applying initial configuration: Aug 16 09:16:38 INFO LOG: Initializing SX log with STDOUT as output file.
*** Could not find first target for layout STD


Starting internal_startup: [ OK ]
Starting tc_ingress_policy: mDNS policing rate=4000kbit burst=400k
Ingress policing enable on interface mgmt0 rate=9000kbit burst=900k
Ingress policing enable on interface mgmt1 rate=9000kbit burst=900k
[ OK ]
Starting clean_issnvram: Deleting issnvram.txt
[ OK ]
Starting intr_hndl: Starting :
Loading int handler module - Success
[ OK ]
Starting iss-nvram-mac: [ OK ]
Starting copy_rh_files_to_vtmp: [ OK ]
Starting secure_mode_check: [ OK ]
Starting sx_pra: Starting proxy arp management:
Loading proxy arp management module - Success
[ OK ]
Starting udevd: Reloading udev...
[ OK ]
Starting pm: [ OK ]
Starting oops_dump_reg: Starting kernel reg dump:
Loading kernel reg dump module - Success
[ OK ]
Starting lnpuppetvar.sh: [ OK ]
Starting mst: Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
[ OK ]

BUT with some error...
It seems that there are some problems with modification of do_rename_ifs in Step 4

comment out the line:
do_rename_ifs
with:
# do_rename_ifs
and add under it simply:
${NAMEIF}

And the switch is not functioning normally yet
The switch system shutdown & initiated nearly every several minutes

Mellanox Switch
MSX6012-MH [standalone: master] >
System shutdown initiated -- logging off.
Mellanox MLNX-OS Switch Management
MSX6012-MH login:

I'm going to test the conversion with version 3.6.1002 instead of 3.6.8012 later.

 

[klui]

You have a typo in rename_ifs but it's difficult to get wrong.

Look at your changes in the file as well as /etc/mactab where I used lower case octets. My logs don't show "Renaming: no changes required"; rather it prints "Running renaming interfaces."

 

[klui]

For those who converted their SX6012 running 3.6.8012, does the cli allow you to (all lower case) issue the following commands?

enable​

ip l3​

show vlan​

show protocol​

config t​

vlan 10​

Mine doesn't allow me to issue commands in red. I can add VLANs and make changes in the webUI.

show vlan​

config t​

vlan 10​

show protocol doesn't show ethernet protocols. Under 3.6.1002 ethernet protocols are listed.

When I edited customer_rootflop.sh, I set

system_profile="3"​

Thanks!

 

[deff]

Is it possible to reflash SX6790 (PSID: EMC0000000001) to make it more useful?
I'm totally ok it's unmanaged and IB only. What bothers me that it's just QDR, while I wish I could juice FDR out of it.
Made a couple of screenshots from a network manager:


Somebody has already tried to figure it out on the forum, but got no real answer.

 

[wangrb]

Thanks a lot.

I met the same problem with using IB to flash FW as Post#337
It seems like not a good idea to try.


Yeah, I did setenv rootdev /dev/mtdblock7 & saveenv
That's why I got stuck in GUIDEv1.2 Step 2 when trying to redo the conversion
there need a setenv rootdev /dev/mtdblock6 before run mlxlinux


I managed to Flash MLNX-OS firmware following Post#372
It seems no need to mess with /etc/passwd file.

Now I got the MLNX-OS booted

BUT with some error...
It seems that there are some problems with modification of do_rename_ifs in Step 4


And the switch is not functioning normally yet
The switch system shutdown & initiated nearly every several minutes

I'm going to test the conversion with version 3.6.1002 instead of 3.6.8012 later.

Maybe caused by one of the two reasons followed:
1.when using 8012, don't replace the ibd file, just replace chad and hwd.
2.wrong firmware of the switch IC. Use MT_1270110020.bin for flashing.

Before trying this, the switch will logout periodically, just like what you said.
Now it is working properly.

 

[deff]

So... I've chosen LSE1010310021 MSX6025F-xxxS_LS_Ax SwitchX-2 based FDR InfiniBand 1U Switch; 36 QSFP+ ports; unmanaged switch with support of mirroring and multi SW firmware for my SX6790 and successfully bricked it. Instead of 4xQDR I got 0xFDR.
Are there any manuals how to bring it back to life or just toss it into a dumpster?

 

[Gabriel Mateiciuc]

So... I've chosen LSE1010310021 MSX6025F-xxxS_LS_Ax SwitchX-2 based FDR InfiniBand 1U Switch; 36 QSFP+ ports; unmanaged switch with support of mirroring and multi SW firmware for my SX6790 and successfully bricked it. Instead of 4xQDR I got 0xFDR.
Are there any manuals how to bring it back to life or just toss it into a dumpster?

I've bricked a 6015 this way ... see if you have a console port to get to some kind of bootloader, or try and see if it has a recovery switch to try and flash it over IB. The guide here for 6xxx managed switches could provide you with valuable hints for what to try and how.

 

[metag]

Just a shout out to all the contributors to this thread. My first 6012 conversion was done about a year ago. Last week I decided to do the conversion on my second one. The impatient me glanced through the last few pages, grabbed the latest guide and converted it to 3.6.8012. It was smoothly done and the switch was up and running. I then messed it up by switching to the eth profile. I had to redo the conversion. So I decided to go back to where I left off on this thread and see what else I have missed. Oh boy, there are so many of them: the progress on fan mod (both 1002 and 8012), the fae hidden command, the single user mode and also the chroot trick to flash the firmware on switch. You guys are amazing, I learned a lot! Thank you!

 

[metag]

Try this

syntax:
mdreq action /system/chassis/actions/set-fan-speed fan_module string {module_name} fan_number int8 {fan_number} fan_speed int8 {fan_speed} set_max uint8 {max_speed}

example:
mdreq action /system/chassis/actions/set-fan-speed fan_module string "/MGMT/FAN1" fan_number int8 1 fan_speed int8 27 set_max uint8 50

I confirmed this works:

MSX6012S1 [standalone: master] # show fan
-----------------------------------------------------
Module          Device          Fan  Speed     Status
                                     (RPM)
-----------------------------------------------------
MGMT            FAN1            F1   7950.00   OK
MGMT            FAN2            F1   8310.00   OK
MGMT            FAN3            F1   7950.00   OK
MGMT            FAN4            F1   8130.00   OK
MSX6012S1 [standalone: master] # exit
[admin@MSX6012S1 ~]# /opt/tms/bin/mdreq action /system/chassis/actions/set-fan-speed fan_module string "/MGMT/FAN1" fan_number int8 1 fan_speed int8 27 set_max uint8 50
[admin@MSX6012S1 ~]# /opt/tms/bin/mdreq action /system/chassis/actions/set-fan-speed fan_module string "/MGMT/FAN2" fan_number int8 1 fan_speed int8 27 set_max uint8 50
[admin@MSX6012S1 ~]# /opt/tms/bin/mdreq action /system/chassis/actions/set-fan-speed fan_module string "/MGMT/FAN3" fan_number int8 1 fan_speed int8 27 set_max uint8 50
[admin@MSX6012S1 ~]# /opt/tms/bin/mdreq action /system/chassis/actions/set-fan-speed fan_module string "/MGMT/FAN4" fan_number int8 1 fan_speed int8 27 set_max uint8 50
[admin@MSX6012S1 ~]# /opt/tms/bin/cli
MSX6012S1 [standalone: master] > show fan
-----------------------------------------------------
Module          Device          Fan  Speed     Status
                                     (RPM)
-----------------------------------------------------
MGMT            FAN1            F1   4530.00   OK
MGMT            FAN2            F1   4860.00   OK
MGMT            FAN3            F1   4740.00   OK
MGMT            FAN4            F1   4530.00   OK
MSX6012S1 [standalone: master] >
MSX6012S1 [standalone: master] > show temperature
---------------------------------------------------------
Module      Component              Reg  CurTemp    Status
                                        (Celsius)
---------------------------------------------------------
MGMT        SX                     T1   35.00      OK
MGMT        QSFP_TEMP1             T1   32.50      OK
MGMT        QSFP_TEMP2             T1   34.00      OK
MGMT        QSFP_TEMP3             T1   35.50      OK
MGMT        BOARD_MONITOR          T1   40.00      OK
MGMT        CPU_BOARD_MONITOR      T1   43.00      OK
MGMT        CPU_BOARD_MONITOR      T2   69.00      OK

I did not record it before running the command. The temperature of the Monitor2 was about 63 Celsius.

The change did not stick after a reboot though. So, I assume there is a way to run those commands at the startup then we don't need to modify the tc binary?

 

[klui]

The change did not stick after a reboot though. So, I assume there is a way to run those commands at the startup then we don't need to modify the tc binary?

You could try booting into mtd6 and enter the commands in /mnt/root2/etc/initi.d/rcS and placing commands there. You might need to wait until the system stabilizes so some of the daemons are running.

Off chance did you install 8012? Could you check if VLAN commands work per my post at https://forums.servethehome.com/ind...-as-mellanox-sx6xxx-on-ebay.10786/post-279407 ?