Beware of EMC switches sold as Mellanox SX6XXX on eBay

Discussion in 'Networking' started by mpogr, Aug 2, 2016.

  1. llowrey

    llowrey Member

    Joined:
    Feb 26, 2018
    Messages:
    68
    Likes Received:
    41
    Success! ...maybe ...probably... NOT! (See update below)

    I was able to modify the SX6018 .bin firmware image to enable high power transceiver support on ports 1-12. Unfortunately, one of my transceivers has a bad laser so link won't come up but here's what the host side says:

    Code:
            Laser tx bias current (Channel 1)         : 50.094 mA
            Laser tx bias current (Channel 2)         : 50.914 mA
            Laser tx bias current (Channel 3)         : 70.986 mA
            Laser tx bias current (Channel 4)         : 53.386 mA
            Transmit avg optical power (Channel 1)    : 0.7611 mW / -1.19 dBm
            Transmit avg optical power (Channel 2)    : 0.7412 mW / -1.30 dBm
            Transmit avg optical power (Channel 3)    : 0.0000 mW / -inf dBm
            Transmit avg optical power (Channel 4)    : 0.7539 mW / -1.23 dBm
            Rcvr signal avg optical power(Channel 1)  : 2.9937 mW / 4.76 dBm
            Rcvr signal avg optical power(Channel 2)  : 4.0408 mW / 6.06 dBm
            Rcvr signal avg optical power(Channel 3)  : 2.3897 mW / 3.78 dBm
            Rcvr signal avg optical power(Channel 4)  : 3.6674 mW / 5.64 dBm
    
    This shows that it is receiving light on all four channels (wavelengths) from the switch. I was getting all zeros before the firmware mod. Hopefully, when I receive a replacement for the bad transceiver I will be able to declare success completely.

    The firmware .bin is made of a number of section which you can see when using flint's (v)erify command.

    While there is a section called "FW Configuration" that contains a zlib/deflate compressed copy of the ascii ini file it's there for documentation/decoration only and has no effect on the switch. I first tried modding that section and confirmed it using the flint 'dc' command but the change made no difference.

    The real config is in the "Configuration" sections and the ones we're interested in are in the 2nd one. That section contains register settings in the form of 32-bit triplets (address, value, bitmask). When mlxburn generates a firmware from an .mlx file and .ini it apparently parses the .ini and maps the directives to register settings. Without the .mlx it's nearly impossible to tell how to map one to the other.

    Fortunately, the SX6012 has 2 firmwares, one with all ports enabled for high power and the other with just 5 ports. I was able to diff the register settings to figure out which registers, and which values, were responsible for the power setting. Interestingly, the pattern of those registers matched the SX6018 but only for the first 12 ports. The upper 6 ports must be in a separate register block somewhere else with totally different values.

    PM me if you'd like to try this firmware. I was only able to burn the firmware using flint on the switch itself. Doing it from an ib card seemed successful but when the switch rebooted it reverted back. I also tried making an mfa file that could be uploaded via the web ui but that very nearly bricked the switch! Doing it the hard way (flint on the switch) is very reliable and what I recommend.

    Update: I received the replacement transceiver and the switch won't bring up link. I can get link card-to-card with these but not with the switch. The page below seems to indicate that the switches are vendor locked for LR4 modules. Perhaps the Cisco BiDi modules will work because they aren't LR4.

    https://community.mellanox.com/s/article/lr4-support-on-mellanox-switches
     
    #241
    Last edited: Jan 25, 2019
  2. llowrey

    llowrey Member

    Joined:
    Feb 26, 2018
    Messages:
    68
    Likes Received:
    41
    That is very doable. The fan speeds are, fortunately, very easy to find in the register set. But, those values are only the max speed. Do you expect those settings to affect the speeds when they aren't running all-out?

    PM me if you would like me to mod a firmware for you to try.
     
    #242
  3. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    100
    Likes Received:
    8
    I am not sure ,read the msx6005 firmware changelog found some thing interest:
    Firmware Changes in Rev 9.3.2000
    •Lowered the externally managed switches default fan speed from 80% to 60%

    so i download 9.3.0000 9.3.2000 9.4.2000 also sx6012 firmware and extract configure to compare ,maybe this difference is the clue how to modify the firmware?

    1 9.3000
    [fan]
    fan_en=1
    fan0_pwm_duty_cycle=0x1a
    fan0_mode = PWM_OUT
    fan1_mode = TACHO_IN
    fan2_mode = NOT_USED
    fan3_mode = TACHO_IN
    fan4_mode = NOT_USED
    fan5_mode = TACHO_IN
    fan6_mode = NOT_USED
    fan7_mode = TACHO_IN
    fan8_mode = NOT_USED
    fan9_mode = NOT_USED
    fan_group0_present_unq = 11
    fan_group0_present_bank = 0
    fan_group0_present_gpio = 0
    fan_group0_good_led_unq = 11
    fan_group0_good_led_bank = 0
    fan_group0_good_led_gpio = 4
    fan1_hi_limit_rpm = 21000
    fan3_hi_limit_rpm = 21000
    fan5_hi_limit_rpm = 21000
    fan7_hi_limit_rpm = 21000


    2 9.3.2000 (change 80% to 60% fan speed)
    [fan]
    fan_en=1
    fan0_mode = PWM_OUT
    fan1_mode = TACHO_IN
    fan2_mode = NOT_USED
    fan3_mode = TACHO_IN
    fan4_mode = NOT_USED
    fan5_mode = TACHO_IN
    fan6_mode = NOT_USED
    fan7_mode = TACHO_IN
    fan8_mode = NOT_USED
    fan9_mode = NOT_USED
    fan_group0_present_unq = 11
    fan_group0_present_bank = 0
    fan_group0_present_gpio = 0
    fan_group0_good_led_unq = 11
    fan_group0_good_led_bank = 0
    fan_group0_good_led_gpio = 4
    fan1_hi_limit_rpm = 23000
    fan3_hi_limit_rpm = 23000
    fan5_hi_limit_rpm = 23000
    fan7_hi_limit_rpm = 23000
    fan0_pwm_duty_cycle= 103
    fan1_pwm_duty_cycle= 103
    fan2_pwm_duty_cycle= 103


    3 9.4.2000
    [fan]
    fan_en=1
    fan0_mode = PWM_OUT
    fan1_mode = TACHO_IN
    fan2_mode = NOT_USED
    fan3_mode = TACHO_IN
    fan4_mode = NOT_USED
    fan5_mode = TACHO_IN
    fan6_mode = NOT_USED
    fan7_mode = TACHO_IN
    fan8_mode = NOT_USED
    fan9_mode = NOT_USED
    fan_group0_present_unq = 11
    fan_group0_present_bank = 0
    fan_group0_present_gpio = 0
    fan_group0_good_led_unq = 11
    fan_group0_good_led_bank = 0
    fan_group0_good_led_gpio = 4
    fan1_hi_limit_rpm = 23000
    fan3_hi_limit_rpm = 23000
    fan5_hi_limit_rpm = 23000
    fan7_hi_limit_rpm = 23000
    fan0_pwm_duty_cycle= 103
    fan1_pwm_duty_cycle= 103
    fan2_pwm_duty_cycle= 103

    4 sx6012

    [fan]
    fan_en=1
    fan0_mode = PWM_OUT
    fan1_mode = TACHO_IN
    fan2_mode = NOT_USED
    fan3_mode = TACHO_IN
    fan4_mode = NOT_USED
    fan5_mode = TACHO_IN
    fan6_mode = NOT_USED
    fan7_mode = TACHO_IN
    fan8_mode = NOT_USED
    fan9_mode = NOT_USED
    fan1_hi_limit_rpm = 23000
    fan3_hi_limit_rpm = 23000
    fan5_hi_limit_rpm = 23000
    fan7_hi_limit_rpm = 23000
     
    #243
  4. Tim111

    Tim111 New Member

    Joined:
    Jan 9, 2019
    Messages:
    3
    Likes Received:
    0
    Just bought an hp sx6036 can update firmware from hp no problem. Problem is i can't use conversion guide to make all options available.
    I can get to boot manager and USB AND U-BOOT need a password. So unless someone has a way to get in I would not buy re-brand switches.
    The only way i can see to get past this is a password or b someone can create image file that can be flashed. So if you buy HP brand you will only be able to IB and no Ethernet or unless someone knows how to get licence key.
     
    #244
  5. wdavid

    wdavid New Member

    Joined:
    Mar 30, 2018
    Messages:
    1
    Likes Received:
    1
    I would like to report another successful conversion of a SX6012 switch. Many thanks to the great work and generosity of @mpogr .
     
    #245
    RageBone likes this.
  6. JSLEnterprises

    JSLEnterprises New Member

    Joined:
    May 4, 2018
    Messages:
    20
    Likes Received:
    21
    I bought one about a month ago because I couldn't get the firmware flashed on one of my 6012's.
    It reads exactly as listed - MSX6036F-2SFR and while it is HP, its full blown Mellanox. The reason you can find them that cheap, is because Mellanox just EOL'd the switch-x2 platform.

    Last MLXOS version is 3.6.8010 for these guys (I also have access to the Mellanox support site through work where I originally picked up my 2x 6012's)

    They're all running 3.2.0330-15 in B, and -100 in A and are IB only (There are no ethernet/gateway licenses embedded on them to switch them to the VPI switch Profile, they're vanilla 6036F, if you want embedded gateway licence which enables ethernet, look for 6036G models).

    If you already have a 6012 that you've done the conversion to, you can swap the PowerPC board inside each chassis, and you'll now have a SX6036F-2SFS running 3.6.1002 with everything 'unlocked'. The serials of these 6036's start with IL2xxxxxx, as you can see in the attached screen caps, you can see in capture 2, the host id starts with MT132 - that's the serial of the EMC 6012 whose firmware I couldn't flash, but the conversion was completed, everything works with the board swap. The only issue is that the power readings show max power of the psu's being 250w when in reality they're 500w..

    Or you can spend a day rolling up the firmware (7 firmware update installs required) and getting your hands on a gateway license (which strangely is still ridiculously priced if you don't have other channels available to you)
     

    Attached Files:

    #246
    Last edited: Jan 30, 2019
    punkerman and mpogr like this.
  7. JSLEnterprises

    JSLEnterprises New Member

    Joined:
    May 4, 2018
    Messages:
    20
    Likes Received:
    21
    The reason the conversion guide works on EMC 6012's is because the uboot console isn't locked out since EMC does ALL the configuration through that. The locked uboot from Mellanox is by design, because they make their money off the licenses.
     
    #247
    mpogr likes this.
  8. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    100
    Likes Received:
    8
    anyone have 6012 fan pictures and model number?
    now I try mod fan follow im2geek4you method,but need the fan model number so I can search fan data sheet to determine best voltage which can balance noise and temperature.
    @im2geek4you
     
    #248
  9. herby

    herby Active Member

    Joined:
    Aug 18, 2013
    Messages:
    168
    Likes Received:
    44
    This is one of the stock fans from my SX6012:
    [​IMG]
     
    #249
    Freebsd1976 likes this.
  10. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    100
    Likes Received:
    8
    thank you very much . btw did you mod or replace the fan for lower noise ?
     
    #250
  11. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,752
    Likes Received:
    615
    I actually have to say, when the switch is booted I don't think that the noise is too bad.
    Booting o/c is annoying, but then... o/c I don't sit next to it...
     
    #251
  12. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,752
    Likes Received:
    615
    I actually have to say, when the switch is booted I don't think that the noise is too bad.
    Booting o/c is annoying, but then... o/c I don't sit next to it...
     
    #251
  13. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    100
    Likes Received:
    8
    unfortunately I sit near by the switch , which is very annoying.
    but I search FFB0412UHN-TD34 ,there is no result.
    search on delta-fan.com found similar FFB0412UHN-SP4Z , which V.Range is 7.0V~13.2V ,so maybe low fan voltage to 5v is not an good idea, but @@im2geek4you said 5v is good for he.

    ps: I am wrong , the similar fan model is FFB0412UHN-C , which V.Range is 5.5V~13.2V , so I will set input voltage 6v to the fan
     
    #252
    Last edited: Feb 1, 2019
  14. RageBone

    RageBone Active Member

    Joined:
    Jul 11, 2017
    Messages:
    271
    Likes Received:
    73
    Why not replace them with noctua fans?
     
    #253
    Last edited: Feb 1, 2019
  15. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    100
    Likes Received:
    8
    there two reasons:
    1 according someone in another thread, for quiet need pull out the noctua fan's pwm wire ,otherwise it will often reinitiate which will make noise.
    2 noctua fan is a bit expensive , four noctua fans will cost half price of the sx6012.
     
    #254
  16. herby

    herby Active Member

    Joined:
    Aug 18, 2013
    Messages:
    168
    Likes Received:
    44
    I actually bought mine from another forum member that did im2geek4you's mod. I tried the switch that way; but I really can't stand the pitch of 40mm fans that move any reasonable amount of air. In the end I'm ashamed to admit I went with the same brutal methods I took with my Rackable SE-3016 disk shelf.

    So my switch now has two 80mm fans mounted to the top in the back and the rear vents covered. I know the air flow is high enough and its quiet; however I'm sure my switch now has next to no resale value and the 1U above it is only free for short depth stuff.
     
    #255
  17. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    100
    Likes Received:
    8
    Oh , i used to consider mount two 14cm fan pull from dell desktop pc , but no circle saw stop me.
     
    #256
  18. llowrey

    llowrey Member

    Joined:
    Feb 26, 2018
    Messages:
    68
    Likes Received:
    41
    Pro tip: Enabling jumbo frames can cut the tftp image download time significantly. For me it was 30min with default settings and a little less than 5min with MTU 9000.

    Code:
    ifconfig eth0 mtu 9000
    tftp -b 8192 -gr <imagefullpath> <yourtftpserverip>
    
    My example above tells tftp to transfer 8k per packet. This cuts the ping-pong latency quite a bit.
     
    #257
  19. efschu2

    efschu2 Member

    Joined:
    Feb 14, 2019
    Messages:
    68
    Likes Received:
    9
    Actualy google should be my friend, but it's not :rolleyes:

    Btw much thanks to everyone here in bringing so much useful information together!!

    --- edit

    After a lot of reading I'm still confused if I need RoCE v1 or v2 for my iSER HA setup, the question is mainly about choosing CX3 or CX3 Pro.

    In a "stupid" iSCSI setup the failover host would just takeover the IP from the main host and life is going on.
    But with iSER, in my understanding, I have to give the failover-host-NIC the mac address and ip from the main-host-NIC and it should do the same? Do I need RoCE v1 or v2 for this? All iSER initiators (4-5) and the iSER HA target will be on the same switches.

    Too much non-informative information out there about RoCE and iSER, need to stop reading now cus my head starts to smoke.

    All example iSER RoCE v1 setups I found were using point to point connection w/o a switch and forsure w/o multiple initiators nor a failover setup.

    I think I just need RoCE v1 but I'd like to be sure about it. Buying twice is always a shame.
     
    #258
    Last edited: Feb 21, 2019
  20. mpogr

    mpogr Active Member

    Joined:
    Jul 14, 2016
    Messages:
    113
    Likes Received:
    75
    iSER initiator setup for VSphere 6.7 is very straightforward and doesn't deal with any particulars of RoCE, as it's all high level. Look up VMware documentation for this.
    What are you planning to use as your iSER target? I use SCST on CentOS (over ZFS) and, again, I don't need to deal with any RoCE particulars when setting it up. I'm in the process of swapping my cards with CX3 Pro instead of CX3 (already swapped on Linux, but not on VMware yet), so will report the findings once the full swap is done...
     
    #259
  21. efschu2

    efschu2 Member

    Joined:
    Feb 14, 2019
    Messages:
    68
    Likes Received:
    9
    My question is more about an active/passive HA solution with dual ported disks, so if the primary-node gets an issue, the ZFS pool will be imported on the failover-node and then representing the LUN. Absolutly no problem with iSCSI because it's only TCP/IP so pacemaker takes care of the IP, have done a few setups like this, unlike iSCSIs IP configuration on iSER RoCE v1 this could only be done by mac address "overwrite" (in my understanding). And with RoCE v2 I could handle that by IP, like in a "normal" iSCSI environment?
    If you have some time left, could you test this?

    If the failover could be handled by mac address overwrite then a CX3 would be enough (I see some IBM ones for 25$), if not I have to take CX3 Pros (which almost costs ten times as much).
     
    #260

Share This Page