Beware of EMC switches sold as Mellanox SX6XXX on eBay

Discussion in 'Networking' started by mpogr, Aug 2, 2016.

  1. meltman

    meltman New Member

    Joined:
    Jan 29, 2017
    Messages:
    1
    Likes Received:
    0
    I want to recognize mpogr for his excellent conversion guide.

    I'm now the proud owner of a fully functional Mellanox SX6012 switch.

    A bit more info about the switch - once it's fully up on MLNX-OS it is quite tame noise wise. The fans are more than insane while booting or outside of a fan controlled environment.

    The switch has active power management and reporting. My switch is reporting between 30-40w energy use with four 10Gb SFP+ modules in QSFP+ adapters.

    I'm only using it as an L2 ethernet switch as of now.

    Cheers!
     
    #21
  2. Homegamer

    Homegamer New Member

    Joined:
    Feb 11, 2018
    Messages:
    1
    Likes Received:
    0
    @mpogr Hey, do you still have the guide for flashing the MLNX-OS?
    Would be great if you could help me with the conversion.

    I sent you a PM!
     
    #22
  3. quovadisza

    quovadisza New Member

    Joined:
    Feb 10, 2018
    Messages:
    1
    Likes Received:
    0
    Tried to PM you with no luck. Anyone have any details on the conversion?
     
    #23
  4. mpogr

    mpogr Active Member

    Joined:
    Jul 14, 2016
    Messages:
    113
    Likes Received:
    75
    Guys, I'm on a holiday (in The Bahamas :) ) and have just received a tremendous amount of PMs asking for the guide. Please, be patient, I'll answer each and every one of you when I have time. Cheers!
     
    #24
    _alex and Marsh like this.
  5. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,590
    Likes Received:
    543
    Lucky you - happy holidays :cool:
     
    #25
  6. llowrey

    llowrey Member

    Joined:
    Feb 26, 2018
    Messages:
    56
    Likes Received:
    31
    I am eyeing an SX6018 and am hoping @mpogr's procedure will work. But, worst case, if the EMC SX6012 can be flashed with the SX6005 firmware for unmanaged use does that mean that the SX6018 should be able to be flashed with the SX6015 firmware? So, at worst, I end up running opensm somewhere and have an IB only network using the SX6018. Correct?
     
    #26
  7. mpogr

    mpogr Active Member

    Joined:
    Jul 14, 2016
    Messages:
    113
    Likes Received:
    75
    All correct, however, I don't see any reason why you wouldn't be able to convert it to proper SX6018. A few people have already done that.
     
    #27
  8. llowrey

    llowrey Member

    Joined:
    Feb 26, 2018
    Messages:
    56
    Likes Received:
    31
    I can report another successful conversion, thanks to @mpogr. I now have a fully functional SX6018. I'm not doing anything fancy atm (no infiniband) but am exercising a few interesting bits of functionality.

    I have one CX3 connected via a no-name DAC. The switch allows me to select 56GbE but I get no link when I do. From what I understand, my DAC is probably coded for 40Gb max so I need to chase down a DAC rated for 56Gb (FDR).

    I have a Dell branded Avago transceiver hooked up to 4 different hosts via an MPO->4xLC breakout cable from FS and a no-name QSFP+->4xSFP+ DAC breakout going to 4 hosts. Splitting the ports had to be done via the CLI but the split ports can be managed in the web UI.

    I'm also running several VLANs and all pass through to my other switch just fine.
     
    #28
    Mithril and _alex like this.
  9. llowrey

    llowrey Member

    Joined:
    Feb 26, 2018
    Messages:
    56
    Likes Received:
    31
    FYI, I finally acquired an FDR cable (Mellanox) and am now able to connect at 56GbE. I only have one 56GbE device so I can't verify the speed but both the switch and card show a link speed of 56GbE.
     
    #29
  10. llowrey

    llowrey Member

    Joined:
    Feb 26, 2018
    Messages:
    56
    Likes Received:
    31
    I seem to have a problem with setting the LAG hash mode. According to the manual, more than one criteria can be specified. But, when I check more than one box on the web UI only last one remains checked after the change is applied. The same thing happens when I do the same via the cli. Bug?

    Code:
    ibswitch [standalone: master] (config) # port-channel load-balance ethernet source-destination-ip source-destination-port
    ibswitch [standalone: master] (config) # show interfaces port-channel load-balance
    source-destination-port
    ibswitch [standalone: master] (config) #
    
     
    #30
  11. jak

    jak New Member

    Joined:
    Apr 17, 2016
    Messages:
    6
    Likes Received:
    0
    First I would like to thank @mpogr for conversion guide.

    I currently have two SX6012 connected together with a Mellanox FDR cable. The link between the two switches does not come up if I set the speed to 56 GbE through the web interface or when I force the interface speed in the command line. This is all done in the ethernet profile. Am I doing something wrong?
     
    #31
  12. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,590
    Likes Received:
    543
    Have you tried setting 40 for testers?
     
    #32
  13. jak

    jak New Member

    Joined:
    Apr 17, 2016
    Messages:
    6
    Likes Received:
    0
    Yes the switches work just fine at 40 gigabits.
     
    #33
  14. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,590
    Likes Received:
    543
    You sure that the cable is FDR? And the correct license is active?
     
    #34
  15. jak

    jak New Member

    Joined:
    Apr 17, 2016
    Messages:
    6
    Likes Received:
    0
    Both switches have the 56 GbE license active and they are connected over an FDR cable.
    [​IMG]

    [​IMG]
     
    #35
  16. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,590
    Likes Received:
    543
    Does it work with NICs?
    Can you try IB?
     
    #36
  17. llowrey

    llowrey Member

    Joined:
    Feb 26, 2018
    Messages:
    56
    Likes Received:
    31
    I've noticed that, for my one linux box that has a 56GbE card (CX3), it doesn't show link until about 60s after my 1GbE NIC does. It's solid once it's up but there seems to be a delay that I don't get when the port is set to 40GbE.
     
    #37
  18. thfs1002

    thfs1002 New Member

    Joined:
    Mar 24, 2018
    Messages:
    2
    Likes Received:
    0
    Hi, all.

    As I just simply can't find the SX6012 firmware, so I'd like to flash EMC6012 into MLNX 6005.

    So, I'd like to flash the firmware into my emc sx6012 by issuing below:

    flint –-allow_psid_change –d lid-0x<yourswitchlid> -i ./<firmwarename>.bin b

    Where firmwarename is fw-sx-x_y_zzzz-MSX6005F_Ax.bin. Is it ok?
     
    #38
  19. jak

    jak New Member

    Joined:
    Apr 17, 2016
    Messages:
    6
    Likes Received:
    0
    The server I am using has a Mellanox ConnectX 3 adapter with the OFED driver on ubuntu 16.04. Connecting to the switch in ethernet mode exhibits will only connect using 40 GbE and waiting does not help. Connecting to the switch in VPI or IB mode using infiniband it will only connect at FDR10 speeds. When I look at the switch all the ports say their supported speed maxs out at FDR10 instead of FDR. The switch will not allow me to force FDR to be a supported speed on any of the ports.
     
    #39
  20. mpogr

    mpogr Active Member

    Joined:
    Jul 14, 2016
    Messages:
    113
    Likes Received:
    75
    Did you try forcing the FDR speed using the iboportstate trick described in the guide (in IB mode)?

    Sent from my SM-G935F using Tapatalk
     
    #40

Share This Page