Beware of EMC switches sold as Mellanox SX6XXX on eBay

[jaydee63]

I've greatly appreciated the conversion guide and advice in this thread.

For the fan speed mod, the 27% value still caused speed ramping a couple of times a minute, on this dual-PSU SX6012 specimen. 28% (0x1C) -- patched into the "tc" file in the 6 places noted -- worked OK with the following (very quiet) outcome:

BlitzenSwitch [standalone: master] > sh fan
=====================================================
Module Device Fan Speed Status
(RPM)
=====================================================
MGMT FAN1 F1 4650.00 OK
MGMT FAN2 F1 4650.00 OK
MGMT FAN3 F1 4470.00 OK
MGMT FAN4 F1 4380.00 OK

BlitzenSwitch [standalone: master] > sh temperature
========================================================
Module Component Reg CurTemp Status
(Celsius)
========================================================
MGMT BOARD_MONITOR T1 30.00 OK
MGMT CPU_BOARD_MONITOR T1 31.00 OK
MGMT CPU_BOARD_MONITOR T2 53.00 OK
MGMT QSFP_TEMP1 T1 28.00 OK
MGMT QSFP_TEMP2 T1 29.50 OK
MGMT QSFP_TEMP3 T1 30.50 OK
MGMT SX T1 31.00 OK​

Basement ambient temperature is quite low, but the switch is unloaded at this time.

I initially tried 32% after 27% thrashed. It worked -- constant lower speed -- but with irritating harmonics (subjectively more objectionable than the original 40%).

I can confirm @up3up4's findings on my SX6012, sweet sweet silence!

ORG21-SW-XLG01 [standalone: master] # sh fan
=====================================================
Module          Device          Fan  Speed     Status
                                     (RPM)
=====================================================
MGMT            FAN1            F1   5130.00   OK
MGMT            FAN2            F1   5400.00   OK
MGMT            FAN3            F1   5130.00   OK
MGMT            FAN4            F1   5130.00   OK
ORG21-SW-XLG01 [standalone: master] # sh temperature
========================================================
Module      Component              Reg  CurTemp   Status
                                        (Celsius)
========================================================
MGMT        BOARD_MONITOR          T1   42.00     OK
MGMT        CPU_BOARD_MONITOR      T1   45.00     OK
MGMT        CPU_BOARD_MONITOR      T2   68.00     OK
MGMT        QSFP_TEMP1             T1   37.00     OK
MGMT        QSFP_TEMP2             T1   38.00     OK
MGMT        QSFP_TEMP3             T1   38.50     OK
MGMT        SX                     T1   51.00     OK

Only edited the tc file to 27%

 

[jaydee63]

I've updated the guide to v1.2 to include fan speed change on SX6012, it's been updated on both Sharepoint and dl.hjemmelab.com - /

Would it be appropriate to upload the patched "tc" files to the archive/SharePoint? I have tc_27, tc_28, and tc_32 instances.

I renamed the original [/mnt/root2]/opt/tms/bin/tc file to tc_40 and then uploaded the other files with their percent designation suffixes. I then did ln -s tc_<min %> tc in that directory, generally following the instructions in the newly amended 1.2 guide Appendix A (i.e., as implied, repeatedly booting in to the minimal Linux system to mount /dev/mtdblock7 in read-write mode).

The files are somewhat large (~3.5 MB); I may eventually clean up and delete the spurious ones.

 

[Necrotyr]

The guide works for other types of the switch, but we've only been able to find the TC values for the SX6012 for now, until we find the values for the 18 and 32 switches I'd rather let people modify the file themselves.

 

[Freebsd1976]

help needed , today want mod fan speed , but can't boot into minios ( boot mnlxos ok ), I run following cmd in u-boot
1 setenv rootdev /dev/mtdblock6
2 setenv image_kernel_args loglevel=2
3 setenv jffs2_args "setenv bootargs root=${rootdev} rootfstype=jffs2 rw reset_button=${reset_button} ${image_kernel_args} ${extra_args}"
4 run mlxlinux

then console show "/bin/sh: can't access tty; job control turned off ", anyone can help me ?

btw this is u-boot env's
=> printenv rootdev
rootdev=/dev/mtdblock6
=> printenv image_kernel_args
image_kernel_args=loglevel=2
=> printenv jffs2_args
jffs2_args=setenv bootargs root=${rootdev} rootfstype=jffs2 ro reset_button=${reset_button} ${image_kernel_args} ${extra_args}
=> setenv jffs2_args "setenv bootargs root=${rootdev} rootfstype=jffs2 rw reset_button=${reset_button} ${image_kernel_args} ${extra_args}"
=> printenv jffs2_args
jffs2_args=setenv bootargs root=/dev/mtdblock6 rootfstype=jffs2 rw reset_button=0 loglevel=2


edit:
it seems "setenv image_kernel_args loglevel=2 " cause the problem , but when i ignore this command, boot into minios is long long time, about half hours and tons of ECC error msg in console . anyone know how to slove this?

 

[up3up4]

help needed , today want mod fan speed , but can't boot into minios ( boot mnlxos ok ), I run following cmd in u-boot
1 setenv rootdev /dev/mtdblock6
2 setenv image_kernel_args loglevel=2
3 setenv jffs2_args "setenv bootargs root=${rootdev} rootfstype=jffs2 rw reset_button=${reset_button} ${image_kernel_args} ${extra_args}"
4 run mlxlinux

then console show "/bin/sh: can't access tty; job control turned off ", anyone can help me ?

btw this is u-boot env's
=> printenv rootdev
rootdev=/dev/mtdblock6
=> printenv image_kernel_args
image_kernel_args=loglevel=2
=> printenv jffs2_args
jffs2_args=setenv bootargs root=${rootdev} rootfstype=jffs2 ro reset_button=${reset_button} ${image_kernel_args} ${extra_args}
=> setenv jffs2_args "setenv bootargs root=${rootdev} rootfstype=jffs2 rw reset_button=${reset_button} ${image_kernel_args} ${extra_args}"
=> printenv jffs2_args
jffs2_args=setenv bootargs root=/dev/mtdblock6 rootfstype=jffs2 rw reset_button=0 loglevel=2


edit:
it seems "setenv image_kernel_args loglevel=2 " cause the problem , but when i ignore this command, boot into minios is long long time, about half hours and tons of ECC error msg in console . anyone know how to slove this?

The 3rd command should be
setenv jffs2_args 'setenv bootargs root=${rootdev} rootfstype=jffs2 rw reset_button=${reset_button} ${image_kernel_args} ${extra_args}'

 

[Freebsd1976]

did you also run "setenv image_kernel_args loglevel=2" or not ?
I try setenv jffs2_args 'setenv bootargs root=${rootdev} rootfstype=jffs2 rw reset_button=${reset_button} ${image_kernel_args} ${extra_args}' with setenv image_kernel_args loglevel=2 , system can't boot .
only setenv jffs2_args 'setenv bootargs root=${rootdev} rootfstype=jffs2 rw reset_button=${reset_button} ${image_kernel_args} ${extra_args}' itselft boot ,but take long long time

 

[up3up4]

did you also run or not ?
setenv image_kernel_args loglevel=2
I try setenv jffs2_args 'setenv bootargs root=${rootdev} rootfstype=jffs2 rw reset_button=${reset_button} ${image_kernel_args} ${extra_args}' with setenv image_kernel_args loglevel=2 , system can't boot .
only setenv jffs2_args 'setenv bootargs root=${rootdev} rootfstype=jffs2 rw reset_button=${reset_button} ${image_kernel_args} ${extra_args}' itselft boot ,but take long long time

I will saveenv for 2nd and 3rd command then reboot. No more headache everytime boot into mini linux.

 

[Freebsd1976]

I will saveenv for 2nd and 3rd command then reboot. No more headache everytime boot into mini linux.

when i run 2nd cmd ,saveenv or not , the system can't boot . not run it, take 50mins to enter minios, don't know why.

 

[Freebsd1976]

my 27% mod on sx6012 one psu , fan speed is different (big mistake ,upload wrong file after whole night try )
=====================================================
Module Device Fan Speed Status
(RPM)
=====================================================
MGMT FAN1 F1 7800.00 OK
MGMT FAN2 F1 7800.00 OK
MGMT FAN3 F1 7500.00 OK
MGMT FAN4 F1 7800.00 OK

this 28% mod on sx6012 one psu , fan speed is ok
=====================================================
Module Device Fan Speed Status
(RPM)
=====================================================
MGMT FAN1 F1 4380.00 OK
MGMT FAN2 F1 4530.00 OK
MGMT FAN3 F1 4530.00 OK
MGMT FAN4 F1 4710.00 OK

 

[Freebsd1976]

4020 EFB0412HHB 65% , mod tc hwd health (max fan speed mod only)
sx6012 [standalone: master] > sh fan
=====================================================
Module Device Fan Speed Status
(RPM)
=====================================================
MGMT FAN1 F1 4800.00 OK
MGMT FAN2 F1 4710.00 OK
MGMT FAN3 F1 4800.00 OK
MGMT FAN4 F1 4710.00 OK

4020 EFB0412HHB70% , mod tc hwd health (max fan speed mod only)
=====================================================
Module Device Fan Speed Status
(RPM)
=====================================================
MGMT FAN1 F1 5340.00 OK
MGMT FAN2 F1 5280.00 OK
MGMT FAN3 F1 5400.00 OK
MGMT FAN4 F1 5400.00 OK

sx6012 [standalone: master] > show temp
========================================================
Module Component Reg CurTemp Status
(Celsius)
========================================================
MGMT BOARD_MONITOR T1 33.00 OK
MGMT CPU_BOARD_MONITOR T1 34.00 OK
MGMT CPU_BOARD_MONITOR T2 56.00 OK
MGMT QSFP_TEMP1 T1 30.50 OK
MGMT QSFP_TEMP2 T1 32.00 OK
MGMT QSFP_TEMP3 T1 33.50 OK
MGMT SX T1 33.00 OK

 

[Eong]

Thanks to everyone here. I got two and converted them. They work perfectly with 10G BiDi modules.
But I can not connect them together using two MC2210511-LR4 modules from FS. Diagnosis doesn't work either.
Is it related to high power in ini file? Need help. All of the ports are running in ETH mode.

 

[Eong]

High power transceivers (eg LR4 and Cisco BiDi) are not supported by the SX6018. Interestingly, the SX6012 supports them on all ports. I verified that (SX6012 and SX6018) with Arista QSFP-40G-UNIV transceivers (LR4 but supporting duplex MMF in addition to SMF).

It should be a simple matter of adding "module_power_level_supported=5" to the "[module#]" sections for each port in the firmware config. The problem is that without the .mlx firmware source it does not appear to be possible to generate a firmware image (.bin) with a custom config.

Questions:

1. Do any of you have, or know where to find, a source .mlx for the SX6018?
2. Is there a way to modify the config entries of a .bin firmware image?
3. Do any of you have a support contract that would allow you to request an .mlx for the SX6018?

I tried the Mellanox MC2210511-LR4 Compatible transceivers but it doesn't work on my SX6012. Could you please confirm if the Arista QSFP-40G-UNIV one works under single mode? I am also wondering if it is related to the power supply. Did you connect both when you test? Thanks a lot.

 

[llowrey]

I tried the Mellanox MC2210511-LR4 Compatible transceivers but it doesn't work on my SX6012. Could you please confirm if the Arista QSFP-40G-UNIV one works under single mode? I am also wondering if it is related to the power supply. Did you connect both when you test? Thanks a lot.

I managed to hack the SX6018 firmware to apply power to high power transceivers (eg LR4 and BiDi) but the switch would not recognize them. I verified that the transceivers were powered and emitting laser light so I think the issue is that the switch firmware won't recognize them because they aren't SR4s. I tried both the Arista QSFP-40G-UNIV and the Cisco BiDi with both my SX6012s and my SX6018. I tried switch-to-switch between the SX6012s and between SX6012 and SX6018. I think the only answer is to find a way to reprogram the transceivers to advertise themselves as SR4.

 

[Eong]

I managed to hack the SX6018 firmware to apply power to high power transceivers (eg LR4 and BiDi) but the switch would not recognize them. I verified that the transceivers were powered and emitting laser light so I think the issue is that the switch firmware won't recognize them because they aren't SR4s. I tried both the Arista QSFP-40G-UNIV and the Cisco BiDi with both my SX6012s and my SX6018. I tried switch-to-switch between the SX6012s and between SX6012 and SX6018. I think the only answer is to find a way to reprogram the transceivers to advertise themselves as SR4.

Thank you. So even with 6012 only, it doesn't work actually? Maybe it only allows LR4 modules from Mellanox? My modules have a wrong code that FS forgot to program them. I'm going to ship them back for programming.

 

[Necrotyr]

@SGS I've now had time to edit the tc file for my SX6018.

I edited all the 00 28 00 0A 00 28 to 00 1C 00 0A 00 1C I also edited the other occurences in the tables (some 3C and 32 values instead of 28).

The fan module quietened down to around 6800rpm, but sadly the PSU fans are ignoring the values in the tc file and stay at 11000rpm.

I wonder if they are located somewhere else.

 

[SGS]

I wonder if they are located somewhere else.

These bytes in hwd and health I posted in #595:
00 00 00 08 00 00 00 1E 00 00 00 08 00 00 32 C8
00 00 00 00 00 00 32 C8

Map to:
.long 8 # chassis_fan_speed (%)
.long 0x1E # spine_fan_speed
.long 8 # ps_fan_speed (%)
.long 0x32C8 # chassis_fan_max_speed
.long 0 # spine_fan_max_speed
.long 0x32C8 # ps_fan_max_speed

(longs on PPC are 32 bit)
I've only modified these values on a 6012 with Noctua fans though and found a working config. It's going to take some experimentation which is going to be frustrated by not knowing for sure which table is correct for the 6018

 

[Necrotyr]

Why is the chassis fan speed only 8%, is that 8% above minimum?

 

[SGS]

Why is the chassis fan speed only 8%, is that 8% above minimum?

That's a good question I don't know the answer to. I don't think it can be 8% above minimum because there's no definition for minimum psu fan speed in the tc tables

 

[Freebsd1976]

@Necrotyr export sx6018 log ， read and analysis maybe help . I read it and found something interesting

 

[Necrotyr]

What log?