Beware of EMC switches sold as Mellanox SX6XXX on eBay

Discussion in 'Networking' started by mpogr, Aug 2, 2016.

  1. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,715
    Likes Received:
    425
    Weird... I tried it a few months ago and got different error messages when I tried to enable the sm manager with the vpi profile.

    Will try this weekend again and report back.
     
    #81
  2. mpogr

    mpogr Active Member

    Joined:
    Jul 14, 2016
    Messages:
    113
    Likes Received:
    75
    You can't enable both SM and Gateway at the same time...
     
    #82
  3. mpogr

    mpogr Active Member

    Joined:
    Jul 14, 2016
    Messages:
    113
    Likes Received:
    75
    The short answer is no, at least not easily. PM me for the longer answer...
     
    #83
  4. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    191
    Likes Received:
    69
    Got mine in today. Holy s*** this thing is DEEP! Anyone got a line on the rack mounting kit?
     
    #84
  5. Hindsight

    Hindsight Member

    Joined:
    Mar 28, 2016
    Messages:
    55
    Likes Received:
    13
    So I bought the one with the "rail kit" it came in the square casing and it had inner rails but no outer rails. I emailed them to ask and they said that's all they had.

    Luckily I have some generic rack shelf rails and so the case slides right in nicely without the inner rails attached.

    https://www.amazon.com/NavePoint-Adjustable-Mount-Server-Shelves/dp/B0060RUVBA
     
    #85
  6. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    191
    Likes Received:
    69
    I got to the point where I need to flash the switch, but now I'm stuck. Is the switch picky about optics? I can see the red light in the QSFP on my card, but none of the ones I plug into the switch light up. I tried running opensm, but it just keeps saying SM port is down.
     
    #86
  7. Hindsight

    Hindsight Member

    Joined:
    Mar 28, 2016
    Messages:
    55
    Likes Received:
    13
    The greatest thing about the Mellanox nics and switches is it doesn't really care what type of transceivers you use.
    run opensm on your pc and then make sure the switch is back into it's regular mode (not linux) to flash.
     
    #87
  8. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    191
    Likes Received:
    69
    Fully booted the upgraded firmware, but still no link or light in the fiber. The only QSFPs I have are the Brocade XBR-000232s and Cisco QSFP-40G-SR-BD.
     
    #88
  9. Hindsight

    Hindsight Member

    Joined:
    Mar 28, 2016
    Messages:
    55
    Likes Received:
    13
    Can you explain what you mean by upgraded firmware? You were able to upgrade the switch through IB? After that unless you are running opensm it won't be on but you don't need it to be. Once you get the full os booted they will be in vpi mode and come online and you can check the port status from there.
     
    #89
  10. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    191
    Likes Received:
    69
    The mlx os part. I can't get it to link over ib to flash it.
     
    #90
  11. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    191
    Likes Received:
    69
    Looking in the fiber, I see the ConnectX3 has red light, but nothing on the switch side.
     
    #91
  12. Hindsight

    Hindsight Member

    Joined:
    Mar 28, 2016
    Messages:
    55
    Likes Received:
    13
    Yeah I don't believe I had lights on either until opensm was running. If you think it might be the cable you can try and get one of these DACs
    Mellanox MC2206130-001 40G QSFP+ DAC Cable | FS.COM
    They also have 56G versions but they are backordered and i've been waiting a few weeks for mine but the 40Gs came quickly.
     
    #92
  13. mpogr

    mpogr Active Member

    Joined:
    Jul 14, 2016
    Messages:
    113
    Likes Received:
    75
    As @Hindsight pointed out, you need to run opensm first. Make sure it's configured properly. The lights will come on then.
    If anything else fails, you can also run flint on the switch itself. PM me for details.
     
    #93
  14. MikeWebb

    MikeWebb Member

    Joined:
    Jan 28, 2018
    Messages:
    87
    Likes Received:
    19
    I might have a play around with this on my 6012. Looks like a good project, well done. PM sent
     
    #94
  15. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    191
    Likes Received:
    69
    Finally got it working after ordering a DAC. Not sure what the issue was since now my other optics seem to work fine in ETH and IB mode.

    Quick tip, since I went through the whole procedure a few times trying to figure out what the issue was.
    Instead of copying the image.tgz into /mnt/var, do mount -t tmpfs tmpfs /mnt/tmp and use it instead. Saves a lot of time and some writes to the flash.
     
    #95
    metag likes this.
  16. Hindsight

    Hindsight Member

    Joined:
    Mar 28, 2016
    Messages:
    55
    Likes Received:
    13
    Least it's working. I went all DAC for my servers and only using fiber to my desktop and other 10gb LCs with the mtp - 4x LC breakout cable. DACs are nice and cheap.
     
    #96
  17. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    191
    Likes Received:
    69
    Is there any way to set up inband management on the ethernet side? I can't seem to find any options to do so. If not I guess I'll just connect mgmt2 to my other switch.
     
    #97
  18. Hindsight

    Hindsight Member

    Joined:
    Mar 28, 2016
    Messages:
    55
    Likes Received:
    13
    http://www.mellanox.com/related-docs/prod_management_software/MLNX-OS_ETH_v3_6_3508_UM.pdf

    Mellanox switches support out-of-band (OOB) dedicated interfaces (e.g. mgmt0, mgmt1) and in-band dedicated interfaces.

    Scroll to 4.1.5 In-Band Management

    Edit - Some of that was blank, This may be better.

    HowTo Enable Inband Management on Mellanox Ethe... | Mellanox Interconnect Community
     
    #98
  19. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    191
    Likes Received:
    69
    Got it, thanks :)
     
    #99
  20. mb300sd

    mb300sd Active Member

    Joined:
    Aug 1, 2016
    Messages:
    191
    Likes Received:
    69
    Is there any way to modify the ini file for the SX6018? I want to use a few Cisco QSFP-40G-SR-BDs in it, but it identifies it as a high power transceiver. LR4 Support on Mellanox Switches | Mellanox Interconnect Community says that it's possible to edit the ini in the SX6012 to enable it but doesn't mention the 6018. Going to pick up some Astria QSFP-40G-UNIV and hope those work for now.
     
    #100

Share This Page