Beware of EMC switches sold as Mellanox SX6XXX on eBay

Discussion in 'Networking' started by mpogr, Aug 2, 2016.

  1. Hindsight

    Hindsight Member

    Joined:
    Mar 28, 2016
    Messages:
    55
    Likes Received:
    13
    At what price? for $39 you can get mellonox branded ones from fs.com
     
    #101
  2. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    191
    Likes Received:
    69
    I need BiDi or UNIV to run 40G over duplex LC fiber. Unfortunately the $39 ones are MPO and my house is wired with LC, did it during a renovation and there's no way to change it without ripping out all the walls again.
     
    #102
  3. Hindsight

    Hindsight Member

    Joined:
    Mar 28, 2016
    Messages:
    55
    Likes Received:
    13
    Ahh that's always fun. If you ever re-do it add some conduit so you can run new stuff easily.
     
    #103
  4. im2geek4you

    im2geek4you Member

    Joined:
    Feb 27, 2017
    Messages:
    30
    Likes Received:
    10
    So, I also manage to modify one of these switches :)
    Thanks mpogr for sharing your knowledge!

    The one thing that really pissed me off was how noisy these little things are!
    Instead of buying new low speed fans which would be just as expensive as the switch, I decided I would have to use the Fans that came with it. I did a lot of experiments and I came up with a solution to keep them quiet without modifying the switch irreversibly. So in order to keep them quiet I decided to lower their voltage thus making them spin slower. A 5 volt linear regulator could do the job except that it would get very hot, and switched regulators are usually big (compared with a 7805).
    But then I found these on ebay:
    10pcs DC-DC 12-24V To 5V 3A Mini Step Down Module Buck Converter Adjustable | eBay

    These small PCB's are cheap as beans and they are the same size as a 7805 plus they allow for variable output (which allows for tuning fan speed to almost any noise desired). I striped 4 noctua NA-RC7 low-noise adapters I had (they are useless with these switches by the way...) and made 4 adapters of them. The small PCB's have a potentiometer that allows to change output voltage. 5V seems to be a good ratio between noise/speed that will allow the switch to operate safelly if you are not going to use it with 12 ports on all the time at full speed.

    I'm getting temps bellow 40ºC on all boards except CPU Board Monitor 2 which is at 52ºC. This is with 4 passive DACs and one QSFP+ adapter connected. PWM and Tacometer will work as well with this mod (although PWM will not make fan speed change a lot at low voltages). Also the switch will complain (you will get an alarm) of low Fan speed. Haven't figured a way of getting rid of this, but it doesn't bother me at all.

    Anyway, just decided to share this, since it might be usefull to someone. Please use this with care, reducing FAN speeds to very low speed/air flow can kill the switch.


    PS: the voltage regulators have an enable pin whitch should be left floating.
    PPS: also another nice thing about this is that power consumption drops by almost 20W, making this switch running at arround 30W... that's insane for the speeds this thing can do.:D

    -Edit- correction regarding Enable pin on these devices.
     

    Attached Files:

    #104
    Last edited: Aug 26, 2018
  5. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    636
    Likes Received:
    211
    So these do VPI (Both IB and Ethernet) after converting to Mellanox's firmware? That price on the sx6012 is really tempting in that case, holy cow...

    Are FDR DACs actually a different pinout or construction from standard QSFP or do the cables just have a capability bit set in firmware? If it's a bit flag it'd be interesting to find a source of cheap DACs that aren't write protected, flip the capability bit and see how they perform.

    That is super slick, nice work! I'm going to keep these in mind the next time I'm quieting a switch.
     
    #105
  6. Hindsight

    Hindsight Member

    Joined:
    Mar 28, 2016
    Messages:
    55
    Likes Received:
    13
    Thanks, That could have been useful for another switch I replaced fans in, will keep it for future. This switch to me isn't that bad when booted into mlnx-os. The EMC boot is extremely loud though.
     
    #106
  7. Luzer

    Luzer New Member

    Joined:
    Mar 1, 2015
    Messages:
    9
    Likes Received:
    1
    another pm sent!
     
    #107
  8. JSLEnterprises

    JSLEnterprises New Member

    Joined:
    May 4, 2018
    Messages:
    20
    Likes Received:
    21
    Anyone having the issue of the management site being stuck "initializing" after successfully doing the conversion?
    I can manage it via CLI, but the management site seems stuck.
     
    #108
  9. Hindsight

    Hindsight Member

    Joined:
    Mar 28, 2016
    Messages:
    55
    Likes Received:
    13
    I think it might have taken a long time (30-40 minutes) I was about to power it off when it came up, after the first time it has always been quick.
     
    #109
  10. JSLEnterprises

    JSLEnterprises New Member

    Joined:
    May 4, 2018
    Messages:
    20
    Likes Received:
    21
    Yeah, still no luck, it might be because I haven't updated the firmware yet. I don't have the ability to update it via inband methods (im preparing my switches via laptop at my other residence, while my homelab is a good 3 hours away.
    is there a method to update it out of band through the management port?

    @Hindsight, does your emc-mellanox switch also have the part number starting 100-886- ?
     
    #110
    Last edited: Aug 29, 2018
  11. Hindsight

    Hindsight Member

    Joined:
    Mar 28, 2016
    Messages:
    55
    Likes Received:
    13
    Yes it does, And you can update the firmware on the switch directly but I haven't tried it. You can pm mpogr for instructions.
     
    #111
  12. Hindsight

    Hindsight Member

    Joined:
    Mar 28, 2016
    Messages:
    55
    Likes Received:
    13
    I finally got my FDR Cables and when I attached them they connect at 40GB. I switch them to 56 GB in the web console and they now show down. VMware only lets me select 40000 for speed. Is there anything else I need to do to get this to work?

    My nics are cx-3
     
    #112
  13. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    636
    Likes Received:
    211
    #113
    JSLEnterprises likes this.
  14. Hindsight

    Hindsight Member

    Joined:
    Mar 28, 2016
    Messages:
    55
    Likes Received:
    13

    Attached Files:

    #114
    JSLEnterprises likes this.
  15. mpogr

    mpogr Active Member

    Joined:
    Jul 14, 2016
    Messages:
    113
    Likes Received:
    75
    @Hindsight this is strange. The only case I had such a problem was when I tried putting CX-3 Pro firmware on my CX-3 (non-Pro) cards. Then they lost 56Gbps speed. But, otherwise, never had issues like this...
     
    #115
  16. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    191
    Likes Received:
    69
    Does anyone know the arguments to the i2c command on the switch? Apparently the BiDi QSFPs I'm using are writable, but only temporarily, and reset on power down. I want to try and send an i2c write in the init script and see if that'll get the switch to activate them.

    Code:
    [admin@SX6018 mlnx_i2c]# i2c 0 r 50 1 0 256
    All parameters are interpreted as hexa values. Application return
    code is zero only when read/write was successfully completed.
    
    [admin@SX6018 mlnx_i2c]# i2c 0 r 50 1 0 256
    
    Usage:
    
            i2c [OPTIONS] <device> <cmd> <i2c_addr> <addr> [<data>]
    
    
    OPTIONS are:
            -h              - Print this message
            -v              - Print Version info
            -a <addr_width> - Set address width to specified value. May
                              be 0, 1, 2 or 4. Default is 1.
            -d <data_width> - Set data width to specified value. May
                              be 1, 2 or 4. Default is 1.
            -x <data_len>   - Present data as two hexa digit string
                              (like 01202040ac). Note that this option is
                              mutually exclusive with "-d" option
    
    Parameters are:
            <device>        - Valid MST device.
            <cmd>           - Command. May be "r[ead]" or "w[rite]".
            <i2c_addr>      - I2C slave address.
            <addr>          - Address (addr_width bytes length) inside I2C
                              target device to read/write operation.
            <data>          - Data (data_width bytes length) to write to
                              target device.
    
    Note that <addr> is ignored if <addr_witdh> == 0
    
     
    #116
  17. JSLEnterprises

    JSLEnterprises New Member

    Joined:
    May 4, 2018
    Messages:
    20
    Likes Received:
    21
    I give up. I cant get any of the ports to initialize (qsfp), so I cant fw update via inband.
    I cant update the fw from the switch, even though flint is literally sitting on the switch ( /dev/mtdblock7/usr/bin/ ), and so is the god damn bin.
     
    #117
    Last edited: Aug 30, 2018
  18. arglebargle

    arglebargle H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈

    Joined:
    Jul 15, 2018
    Messages:
    636
    Likes Received:
    211
    If you're willing to mail it to me instead of tossing it I'll pay postage :p
     
    #118
  19. Hindsight

    Hindsight Member

    Joined:
    Mar 28, 2016
    Messages:
    55
    Likes Received:
    13
    Just give it a day or two break and try some more, it took me 2 days, loud as hell in my room. One of the guys last week had the same problems with port not coming up. Are you running the open subnet manager? ports won't come on without it. He had to buy a DAC before it worked for him.
     
    #119
  20. JSLEnterprises

    JSLEnterprises New Member

    Joined:
    May 4, 2018
    Messages:
    20
    Likes Received:
    21
    Using monoprice dac's (have a qdr IBM switch which my 2 mel/emc switches are going to replace, and the dac's work fine on it. however, it only does l2 ethernet switching when not doing ib which is why I want to swap it out)
    Open Subnet manager is running on one of my servers, connection doesn't come up.
    cards i use are are cx-3 bcct's (which don't do InfiniBand)
     
    #120
    Last edited: Aug 31, 2018

Share This Page