Beware of EMC switches sold as Mellanox SX6XXX on eBay

Discussion in 'Networking' started by mpogr, Aug 2, 2016.

  1. oxynazin

    oxynazin New Member

    Joined:
    Dec 10, 2018
    Messages:
    10
    Likes Received:
    1
    But CX3 supports RoCE according to their product brief pdf. Unclear v1 or v2, but somewhere I saw that CX2 - RoCEv1, CX3 - RoCEv2.
     
    #221
  2. mpogr

    mpogr Active Member

    Joined:
    Jul 14, 2016
    Messages:
    111
    Likes Received:
    69
    Yes, I meant RoCE, and, indeed, CX-3 supports v1, but only CX-3 Pro and up support v2, which is quite a bit of a difference latency-wise.
    I have done some measurements with my setup using SRP a while ago, now need to redo the same baseline to see how things are different with iSER, but, from memory, there was quite a bit of performance loss, especially with multithreaded workloads.
     
    #222
  3. oxynazin

    oxynazin New Member

    Joined:
    Dec 10, 2018
    Messages:
    10
    Likes Received:
    1
    Found this huge article: https://www.demartek.com/Reports_Free/Demartek_IBTA_RoCE_Deployment_Guide_2018-07.pdf Have not read all article, because there're too many new information for me. But found some interesting parts. For SMB Direct there's intersting part: "If RoCE v2 is not necessary for routing purposes, RoCE v1 may give better performance.". And in the end: "The addition of ETS and RoCEv2 Congestion Management resulted in improved RoCE throughput and decreased latency.". As I understood, v2 adds some overhead for Layer3 support but also adds "RoCEv2 Congestion Management (RCM)". So v1 must be at least as fast as v2, until you not utilize network for 100%. Am I right?
     
    #223
  4. zxv

    zxv The more I C, the less I see.

    Joined:
    Sep 10, 2017
    Messages:
    89
    Likes Received:
    30
    #224
  5. manfri

    manfri New Member

    Joined:
    Nov 19, 2015
    Messages:
    26
    Likes Received:
    1
    Kudos to mpogr for the guide... great work... but a bit extreme for my heart, and tooo much effort for my needs...
    I'm gonna pass :cool:))
     
    #225
  6. Tim111

    Tim111 New Member

    Joined:
    Jan 9, 2019
    Messages:
    3
    Likes Received:
    0
    Just signed up, is there a way I can get information for flashing EMC sx6012 please.
     
    #226
  7. Terry Wallace

    Terry Wallace Member

    Joined:
    Aug 13, 2018
    Messages:
    32
    Likes Received:
    7
    If you start at page 1 of this thread and read through it just about all the information has been presented here. :)
    Good luck on conversion.
    Post when you get stuck on a specific step and we'll walk you through it.
     
    #227
  8. Tim111

    Tim111 New Member

    Joined:
    Jan 9, 2019
    Messages:
    3
    Likes Received:
    0
    Thanks Terry.
     
    #228
  9. zxv

    zxv The more I C, the less I see.

    Joined:
    Sep 10, 2017
    Messages:
    89
    Likes Received:
    30
    #229
  10. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,533
    Likes Received:
    355
    #230
  11. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,934
    Likes Received:
    403
  12. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    47
    Likes Received:
    4
    an off-topic question , howto connect sx6012 to internet?
    my sx6012 only for spark cluster in my home lab, no internet access,.
    now for system update which need internet access,I consider connect it to other switch for internet access, my other switch have rj45 1G port, but the question is how to connect two switch , sx6012 only the management port is rj45?
     
    #232
  13. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,533
    Likes Received:
    355
    You could use a qsfp to sfp Adapter and a rj45 sfp Module
     
    #233
  14. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    47
    Likes Received:
    4
    thanks a lot ,I found the other switch has one 10g port, so I think I can buy one qsfp+ to four sfp fan out cable uplink for internet access
     
    #234
  15. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    47
    Likes Received:
    4
    Another question , I know we could not upgrade os ,but could we upgrade the firmware, anyone try this.
     
    #235
  16. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    47
    Likes Received:
    4
    thank a lot , this is want I want. Btw how to do make this output fixed 5v ?the default output seems adjustable .

    PS : find howto fix output ,
    5PCS Mini DC to DC Adjustable Step Down Module 12/24V to 5V 3A Buck Converter Power Supply Module
    There is a YouTube video demo howto do this
     
    #236
    Last edited: Jan 17, 2019
  17. llowrey

    llowrey Member

    Joined:
    Feb 26, 2018
    Messages:
    37
    Likes Received:
    11
    High power transceivers (eg LR4 and Cisco BiDi) are not supported by the SX6018. Interestingly, the SX6012 supports them on all ports. I verified that (SX6012 and SX6018) with Arista QSFP-40G-UNIV transceivers (LR4 but supporting duplex MMF in addition to SMF).

    It should be a simple matter of adding "module_power_level_supported=5" to the "[module#]" sections for each port in the firmware config. The problem is that without the .mlx firmware source it does not appear to be possible to generate a firmware image (.bin) with a custom config.

    Questions:
    1. Do any of you have, or know where to find, a source .mlx for the SX6018?
    2. Is there a way to modify the config entries of a .bin firmware image?
    3. Do any of you have a support contract that would allow you to request an .mlx for the SX6018?
     
    #237
  18. RageBone

    RageBone Member

    Joined:
    Jul 11, 2017
    Messages:
    139
    Likes Received:
    22
    i think that there was a tool to read configs from network-cards, might work with switches too.

    EDIT:
    i think it was one of the few standard tools, mstflint or so.

    Edit:
    http://www.mellanox.com/page/management_tools
    Debug-Utils sounds interesting.
    I think i used MLXBurn before but its to long ago.
    mlxdump, mstdump sound very promising
     
    #238
    Last edited: Jan 19, 2019
  19. llowrey

    llowrey Member

    Joined:
    Feb 26, 2018
    Messages:
    37
    Likes Received:
    11
    I used flint to pull the config from the current firmware image .bins for the SX6018 and the SX6012 which is how I learned that the magic entry is "module_power_level_supported=5". My problem now is how to either load an updated config into the switch or to modify the firmware with a new config.

    As far as I can tell the only way to modify a firmware is to generate a new one using mlxburn and that requires an .mlx source firmware file, which I don't have. I have not seen any tool which will burn just the config.

    It seems Mellanox used to publish firmware sources (.mlx files) on their website, at least for adapter cards, until just a couple of years ago.
     
    #239
  20. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    47
    Likes Received:
    4
    Yeah , if you find a way to modify it , maybe also could set fan speed, this from config.ini for fan control

    [fan]

    fan_en=1

    fan0_mode = PWM_OUT

    fan1_mode = TACHO_IN

    fan2_mode = NOT_USED

    fan3_mode = TACHO_IN

    fan4_mode = NOT_USED

    fan5_mode = TACHO_IN

    fan6_mode = NOT_USED

    fan7_mode = TACHO_IN

    fan8_mode = NOT_USED

    fan9_mode = NOT_USED


    fan1_hi_limit_rpm = 23000

    fan3_hi_limit_rpm = 23000

    fan5_hi_limit_rpm = 23000

    fan7_hi_limit_rpm = 23000
     
    #240

Share This Page