Beware of EMC switches sold as Mellanox SX6XXX on eBay

llowrey

Active Member
Feb 26, 2018
111
75
28
Success! ...maybe ...probably... NOT! (See update below)

I was able to modify the SX6018 .bin firmware image to enable high power transceiver support on ports 1-12. Unfortunately, one of my transceivers has a bad laser so link won't come up but here's what the host side says:

Code:
        Laser tx bias current (Channel 1)         : 50.094 mA
        Laser tx bias current (Channel 2)         : 50.914 mA
        Laser tx bias current (Channel 3)         : 70.986 mA
        Laser tx bias current (Channel 4)         : 53.386 mA
        Transmit avg optical power (Channel 1)    : 0.7611 mW / -1.19 dBm
        Transmit avg optical power (Channel 2)    : 0.7412 mW / -1.30 dBm
        Transmit avg optical power (Channel 3)    : 0.0000 mW / -inf dBm
        Transmit avg optical power (Channel 4)    : 0.7539 mW / -1.23 dBm
        Rcvr signal avg optical power(Channel 1)  : 2.9937 mW / 4.76 dBm
        Rcvr signal avg optical power(Channel 2)  : 4.0408 mW / 6.06 dBm
        Rcvr signal avg optical power(Channel 3)  : 2.3897 mW / 3.78 dBm
        Rcvr signal avg optical power(Channel 4)  : 3.6674 mW / 5.64 dBm
This shows that it is receiving light on all four channels (wavelengths) from the switch. I was getting all zeros before the firmware mod. Hopefully, when I receive a replacement for the bad transceiver I will be able to declare success completely.

The firmware .bin is made of a number of section which you can see when using flint's (v)erify command.

While there is a section called "FW Configuration" that contains a zlib/deflate compressed copy of the ascii ini file it's there for documentation/decoration only and has no effect on the switch. I first tried modding that section and confirmed it using the flint 'dc' command but the change made no difference.

The real config is in the "Configuration" sections and the ones we're interested in are in the 2nd one. That section contains register settings in the form of 32-bit triplets (address, value, bitmask). When mlxburn generates a firmware from an .mlx file and .ini it apparently parses the .ini and maps the directives to register settings. Without the .mlx it's nearly impossible to tell how to map one to the other.

Fortunately, the SX6012 has 2 firmwares, one with all ports enabled for high power and the other with just 5 ports. I was able to diff the register settings to figure out which registers, and which values, were responsible for the power setting. Interestingly, the pattern of those registers matched the SX6018 but only for the first 12 ports. The upper 6 ports must be in a separate register block somewhere else with totally different values.

PM me if you'd like to try this firmware. I was only able to burn the firmware using flint on the switch itself. Doing it from an ib card seemed successful but when the switch rebooted it reverted back. I also tried making an mfa file that could be uploaded via the web ui but that very nearly bricked the switch! Doing it the hard way (flint on the switch) is very reliable and what I recommend.

Update: I received the replacement transceiver and the switch won't bring up link. I can get link card-to-card with these but not with the switch. The page below seems to indicate that the switches are vendor locked for LR4 modules. Perhaps the Cisco BiDi modules will work because they aren't LR4.

https://community.mellanox.com/s/article/lr4-support-on-mellanox-switches
 
Last edited:

llowrey

Active Member
Feb 26, 2018
111
75
28
Yeah , if you find a way to modify it , maybe also could set fan speed, this from config.ini for fan control
That is very doable. The fan speeds are, fortunately, very easy to find in the register set. But, those values are only the max speed. Do you expect those settings to affect the speeds when they aren't running all-out?

PM me if you would like me to mod a firmware for you to try.
 

Freebsd1976

Active Member
Feb 23, 2018
302
44
28
I am not sure ,read the msx6005 firmware changelog found some thing interest:
Firmware Changes in Rev 9.3.2000
•Lowered the externally managed switches default fan speed from 80% to 60%

so i download 9.3.0000 9.3.2000 9.4.2000 also sx6012 firmware and extract configure to compare ,maybe this difference is the clue how to modify the firmware?

1 9.3000
[fan]
fan_en=1
fan0_pwm_duty_cycle=0x1a
fan0_mode = PWM_OUT
fan1_mode = TACHO_IN
fan2_mode = NOT_USED
fan3_mode = TACHO_IN
fan4_mode = NOT_USED
fan5_mode = TACHO_IN
fan6_mode = NOT_USED
fan7_mode = TACHO_IN
fan8_mode = NOT_USED
fan9_mode = NOT_USED
fan_group0_present_unq = 11
fan_group0_present_bank = 0
fan_group0_present_gpio = 0
fan_group0_good_led_unq = 11
fan_group0_good_led_bank = 0
fan_group0_good_led_gpio = 4
fan1_hi_limit_rpm = 21000
fan3_hi_limit_rpm = 21000
fan5_hi_limit_rpm = 21000
fan7_hi_limit_rpm = 21000


2 9.3.2000 (change 80% to 60% fan speed)
[fan]
fan_en=1
fan0_mode = PWM_OUT
fan1_mode = TACHO_IN
fan2_mode = NOT_USED
fan3_mode = TACHO_IN
fan4_mode = NOT_USED
fan5_mode = TACHO_IN
fan6_mode = NOT_USED
fan7_mode = TACHO_IN
fan8_mode = NOT_USED
fan9_mode = NOT_USED
fan_group0_present_unq = 11
fan_group0_present_bank = 0
fan_group0_present_gpio = 0
fan_group0_good_led_unq = 11
fan_group0_good_led_bank = 0
fan_group0_good_led_gpio = 4
fan1_hi_limit_rpm = 23000
fan3_hi_limit_rpm = 23000
fan5_hi_limit_rpm = 23000
fan7_hi_limit_rpm = 23000
fan0_pwm_duty_cycle= 103
fan1_pwm_duty_cycle= 103
fan2_pwm_duty_cycle= 103


3 9.4.2000
[fan]
fan_en=1
fan0_mode = PWM_OUT
fan1_mode = TACHO_IN
fan2_mode = NOT_USED
fan3_mode = TACHO_IN
fan4_mode = NOT_USED
fan5_mode = TACHO_IN
fan6_mode = NOT_USED
fan7_mode = TACHO_IN
fan8_mode = NOT_USED
fan9_mode = NOT_USED
fan_group0_present_unq = 11
fan_group0_present_bank = 0
fan_group0_present_gpio = 0
fan_group0_good_led_unq = 11
fan_group0_good_led_bank = 0
fan_group0_good_led_gpio = 4
fan1_hi_limit_rpm = 23000
fan3_hi_limit_rpm = 23000
fan5_hi_limit_rpm = 23000
fan7_hi_limit_rpm = 23000
fan0_pwm_duty_cycle= 103
fan1_pwm_duty_cycle= 103
fan2_pwm_duty_cycle= 103

4 sx6012

[fan]
fan_en=1
fan0_mode = PWM_OUT
fan1_mode = TACHO_IN
fan2_mode = NOT_USED
fan3_mode = TACHO_IN
fan4_mode = NOT_USED
fan5_mode = TACHO_IN
fan6_mode = NOT_USED
fan7_mode = TACHO_IN
fan8_mode = NOT_USED
fan9_mode = NOT_USED
fan1_hi_limit_rpm = 23000
fan3_hi_limit_rpm = 23000
fan5_hi_limit_rpm = 23000
fan7_hi_limit_rpm = 23000
 

Tim111

New Member
Jan 9, 2019
11
1
1
Just bought an hp sx6036 can update firmware from hp no problem. Problem is i can't use conversion guide to make all options available.
I can get to boot manager and USB AND U-BOOT need a password. So unless someone has a way to get in I would not buy re-brand switches.
The only way i can see to get past this is a password or b someone can create image file that can be flashed. So if you buy HP brand you will only be able to IB and no Ethernet or unless someone knows how to get licence key.
 

JSLEnterprises

New Member
May 4, 2018
20
22
3
39
I bought one about a month ago because I couldn't get the firmware flashed on one of my 6012's.
It reads exactly as listed - MSX6036F-2SFR and while it is HP, its full blown Mellanox. The reason you can find them that cheap, is because Mellanox just EOL'd the switch-x2 platform.

Last MLXOS version is 3.6.8010 for these guys (I also have access to the Mellanox support site through work where I originally picked up my 2x 6012's)

They're all running 3.2.0330-15 in B, and -100 in A and are IB only (There are no ethernet/gateway licenses embedded on them to switch them to the VPI switch Profile, they're vanilla 6036F, if you want embedded gateway licence which enables ethernet, look for 6036G models).

If you already have a 6012 that you've done the conversion to, you can swap the PowerPC board inside each chassis, and you'll now have a SX6036F-2SFS running 3.6.1002 with everything 'unlocked'. The serials of these 6036's start with IL2xxxxxx, as you can see in the attached screen caps, you can see in capture 2, the host id starts with MT132 - that's the serial of the EMC 6012 whose firmware I couldn't flash, but the conversion was completed, everything works with the board swap. The only issue is that the power readings show max power of the psu's being 250w when in reality they're 500w..

Or you can spend a day rolling up the firmware (7 firmware update installs required) and getting your hands on a gateway license (which strangely is still ridiculously priced if you don't have other channels available to you)
 

Attachments

Last edited:
  • Like
Reactions: punkerman and mpogr

JSLEnterprises

New Member
May 4, 2018
20
22
3
39
Just bought an hp sx6036 can update firmware from hp no problem. Problem is i can't use conversion guide to make all options available.
I can get to boot manager and USB AND U-BOOT need a password. So unless someone has a way to get in I would not buy re-brand switches.
The only way i can see to get past this is a password or b someone can create image file that can be flashed. So if you buy HP brand you will only be able to IB and no Ethernet or unless someone knows how to get licence key.
The reason the conversion guide works on EMC 6012's is because the uboot console isn't locked out since EMC does ALL the configuration through that. The locked uboot from Mellanox is by design, because they make their money off the licenses.
 
  • Like
Reactions: mpogr

Freebsd1976

Active Member
Feb 23, 2018
302
44
28
anyone have 6012 fan pictures and model number?
now I try mod fan follow im2geek4you method,but need the fan model number so I can search fan data sheet to determine best voltage which can balance noise and temperature.
@im2geek4you
 
  • Like
Reactions: alexalex

Rand__

Well-Known Member
Mar 6, 2014
5,779
1,292
113
I actually have to say, when the switch is booted I don't think that the noise is too bad.
Booting o/c is annoying, but then... o/c I don't sit next to it...
 

Rand__

Well-Known Member
Mar 6, 2014
5,779
1,292
113
I actually have to say, when the switch is booted I don't think that the noise is too bad.
Booting o/c is annoying, but then... o/c I don't sit next to it...
 

Freebsd1976

Active Member
Feb 23, 2018
302
44
28
I actually have to say, when the switch is booted I don't think that the noise is too bad.
Booting o/c is annoying, but then... o/c I don't sit next to it...
unfortunately I sit near by the switch , which is very annoying.
but I search FFB0412UHN-TD34 ,there is no result.
search on delta-fan.com found similar FFB0412UHN-SP4Z , which V.Range is 7.0V~13.2V ,so maybe low fan voltage to 5v is not an good idea, but @@im2geek4you said 5v is good for he.

ps: I am wrong , the similar fan model is FFB0412UHN-C , which V.Range is 5.5V~13.2V , so I will set input voltage 6v to the fan
 
Last edited:

Freebsd1976

Active Member
Feb 23, 2018
302
44
28
Why not replace Thema with noctua Fans?
there two reasons:
1 according someone in another thread, for quiet need pull out the noctua fan's pwm wire ,otherwise it will often reinitiate which will make noise.
2 noctua fan is a bit expensive , four noctua fans will cost half price of the sx6012.
 

herby

Active Member
Aug 18, 2013
185
52
28
thank you very much . btw did you mod or replace the fan for lower noise ?
I actually bought mine from another forum member that did im2geek4you's mod. I tried the switch that way; but I really can't stand the pitch of 40mm fans that move any reasonable amount of air. In the end I'm ashamed to admit I went with the same brutal methods I took with my Rackable SE-3016 disk shelf.

So my switch now has two 80mm fans mounted to the top in the back and the rear vents covered. I know the air flow is high enough and its quiet; however I'm sure my switch now has next to no resale value and the 1U above it is only free for short depth stuff.
 

Freebsd1976

Active Member
Feb 23, 2018
302
44
28
I actually bought mine from another forum member that did im2geek4you's mod. I tried the switch that way; but I really can't stand the pitch of 40mm fans that move any reasonable amount of air. In the end I'm ashamed to admit I went with the same brutal methods I took with my Rackable SE-3016 disk shelf.

So my switch now has two 80mm fans mounted to the top in the back and the rear vents covered. I know the air flow is high enough and its quiet; however I'm sure my switch now has next to no resale value and the 1U above it is only free for short depth stuff.
Oh , i used to consider mount two 14cm fan pull from dell desktop pc , but no circle saw stop me.
 

llowrey

Active Member
Feb 26, 2018
111
75
28
Pro tip: Enabling jumbo frames can cut the tftp image download time significantly. For me it was 30min with default settings and a little less than 5min with MTU 9000.

Code:
ifconfig eth0 mtu 9000
tftp -b 8192 -gr <imagefullpath> <yourtftpserverip>
My example above tells tftp to transfer 8k per packet. This cuts the ping-pong latency quite a bit.
 

efschu2

Member
Feb 14, 2019
68
12
8
Actualy google should be my friend, but it's not :rolleyes:

Btw much thanks to everyone here in bringing so much useful information together!!

--- edit

After a lot of reading I'm still confused if I need RoCE v1 or v2 for my iSER HA setup, the question is mainly about choosing CX3 or CX3 Pro.

In a "stupid" iSCSI setup the failover host would just takeover the IP from the main host and life is going on.
But with iSER, in my understanding, I have to give the failover-host-NIC the mac address and ip from the main-host-NIC and it should do the same? Do I need RoCE v1 or v2 for this? All iSER initiators (4-5) and the iSER HA target will be on the same switches.

Too much non-informative information out there about RoCE and iSER, need to stop reading now cus my head starts to smoke.

All example iSER RoCE v1 setups I found were using point to point connection w/o a switch and forsure w/o multiple initiators nor a failover setup.

I think I just need RoCE v1 but I'd like to be sure about it. Buying twice is always a shame.
 
Last edited:

mpogr

Active Member
Jul 14, 2016
115
93
28
51
iSER initiator setup for VSphere 6.7 is very straightforward and doesn't deal with any particulars of RoCE, as it's all high level. Look up VMware documentation for this.
What are you planning to use as your iSER target? I use SCST on CentOS (over ZFS) and, again, I don't need to deal with any RoCE particulars when setting it up. I'm in the process of swapping my cards with CX3 Pro instead of CX3 (already swapped on Linux, but not on VMware yet), so will report the findings once the full swap is done...
 

efschu2

Member
Feb 14, 2019
68
12
8
My question is more about an active/passive HA solution with dual ported disks, so if the primary-node gets an issue, the ZFS pool will be imported on the failover-node and then representing the LUN. Absolutly no problem with iSCSI because it's only TCP/IP so pacemaker takes care of the IP, have done a few setups like this, unlike iSCSIs IP configuration on iSER RoCE v1 this could only be done by mac address "overwrite" (in my understanding). And with RoCE v2 I could handle that by IP, like in a "normal" iSCSI environment?
If you have some time left, could you test this?

If the failover could be handled by mac address overwrite then a CX3 would be enough (I see some IBM ones for 25$), if not I have to take CX3 Pros (which almost costs ten times as much).