100 gig ethernet - which non Mellanox Cards?

Discussion in 'Networking' started by Rand__, May 28, 2019.

  1. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,469
    Likes Received:
    503
    Since my Celestica switch is not working with Mellanox CX4's at 100G I am looking for alternative cards to use.

    I could o/c switch to CX5's+ but those are still around $500/card which is a bit too much.

    So I am looking for cheaper alternatives that will run with ESXi, and ideally is not too picky re DACs since I got a lot of Mellanox branded ones;)

    I have seen the T62100-LP-CR - available for $400 (maybe $300 BO) at this time, that could work, but no idea re DAC compatibility.

    I never looked further since I used to be happy with the MLX cards...

    Edit:
    Found this:
    Meanderful: 100G Ethernet NICs - Broadcom joins Mellanox, Cavium's QLogic, and Chelsio

    Edit2:
    Broadcom uses QSFP56 on the Dual Port cards :/
    100Gb Ethernet Network Adapters
     
    #1
    Last edited: May 28, 2019
  2. Labs

    Labs New Member

    Joined:
    Mar 21, 2019
    Messages:
    26
    Likes Received:
    4
    Look on eBay UK for - Dell Intel Omni-Path 100Gb QSFP28 at £180.
    I remember it was an article on STH about them but I cannot find it now.
    I am not sure about compatibility because there is not much info on the net.
    It could be that it works only with Intel QSFP28. Another seller in Germany was selling them for around 160 Euro but didn't buy any since I couldn't find too much info about compatibility.

    Edit: I found the one in Germany also but they are 200 Euro not 160 as I remembered.
    eBay article number: 123255614885
     
    #2
  3. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,469
    Likes Received:
    503
    Yeah i am not sure either how compatible Omni-Path is with anything else tbh..
    But thanks.
     
    #3
  4. Labs

    Labs New Member

    Joined:
    Mar 21, 2019
    Messages:
    26
    Likes Received:
    4
    Yes, that's why I didn't buy any of those cards but might be an alternative solution to your problem or not...
    You can also try some AOC cables or QSFP28 with MPO fiber.
    I never had issues with QSFP28 and MPO fiber interconnecting different switches/routers but I had issues with DAC cables where the DAC didn't have the capabilities identified correctly and was flapping or dropping to 40Gbe. Forcing to 100Gbe was disconnecting the link completely.
     
    #4
  5. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,469
    Likes Received:
    503
    I have tried 100GB IB cable, Ethernet cable, IB Fibre - only dedicated Ethernet fibre is missing since i didnt have any.
    The ones I used were Mellanox certified cables, any of the above worked with CX5's just not CX4's and 100G (they worked with 40G)
     
    #5
  6. necr

    necr New Member

    Joined:
    Dec 27, 2017
    Messages:
    20
    Likes Received:
    11
    #6
  7. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,469
    Likes Received:
    503
    will need to have a look, thanks:)
     
    #7
  8. Labs

    Labs New Member

    Joined:
    Mar 21, 2019
    Messages:
    26
    Likes Received:
    4
    You are right, it seems the omni-path nics are some interconnect cards not supporting IB or ethernet.

    About Mellanox ConnectX ICs it seems they are different between card generations. Do you have any link where is saying CX4 and CX5 use the same IC?
     
    #8
  9. necr

    necr New Member

    Joined:
    Dec 27, 2017
    Messages:
    20
    Likes Received:
    11
    I never said they used same IC, but they use same ports/SerDes. New 100G PHY is called Shomron.
    As I suspected, differences are tiny. @Rand__ I don't have dual port CX4, but I suggest adjusting polarity on lanes 0 and 2 and also setting power cap to 6.
    See diff:

    Code:
    < module.num[0].label_port= 1
    < module.num[0].type= 1
    < module.num[0].m2l.tx.lane[0].serdes= 3
    < module.num[0].m2l.rx.lane[0].serdes= 0
    < module.num[0].m2l.tx.lane[0].polarity= 1
    < module.num[0].m2l.rx.lane[0].polarity= 0
    < module.num[0].m2l.rx.lane[0].local_port= 0
    < module.num[0].m2l.tx.lane[0].local_port= 0
    < module.num[0].m2l.tx.lane[1].serdes= 2
    < module.num[0].m2l.rx.lane[1].serdes= 1
    < module.num[0].m2l.tx.lane[1].polarity= 1
    < module.num[0].m2l.rx.lane[1].polarity= 1
    < module.num[0].m2l.rx.lane[1].local_port= 0
    < module.num[0].m2l.tx.lane[1].local_port= 0
    < module.num[0].m2l.tx.lane[2].serdes= 1
    < module.num[0].m2l.rx.lane[2].serdes= 2
    < module.num[0].m2l.tx.lane[2].polarity= 1
    < module.num[0].m2l.rx.lane[2].polarity= 0
    < module.num[0].m2l.rx.lane[2].local_port= 0
    < module.num[0].m2l.tx.lane[2].local_port= 0
    < module.num[0].m2l.tx.lane[3].serdes= 0
    < module.num[0].m2l.rx.lane[3].serdes= 3
    < module.num[0].m2l.tx.lane[3].polarity= 1
    < module.num[0].m2l.rx.lane[3].polarity= 1
    < module.num[0].m2l.rx.lane[3].local_port= 0
    < module.num[0].m2l.tx.lane[3].local_port= 0
    < module.num[0].smbus_slv_indx= 4
    < module.num[0].module_power_level_supported= 4
    ---
    > module.num[0].label_port= 1
    > module.num[0].type= 1
    > module.num[0].m2l.tx.lane[0].serdes= 3
    > module.num[0].m2l.rx.lane[0].serdes= 0
    > module.num[0].m2l.tx.lane[0].polarity= 0
    > module.num[0].m2l.rx.lane[0].polarity= 0
    > module.num[0].m2l.rx.lane[0].local_lane= 0
    > module.num[0].m2l.tx.lane[0].local_lane= 0
    > module.num[0].m2l.rx.lane[0].local_port= 0
    > module.num[0].m2l.tx.lane[0].local_port= 0
    > module.num[0].m2l.tx.lane[1].serdes= 2
    > module.num[0].m2l.rx.lane[1].serdes= 1
    > module.num[0].m2l.tx.lane[1].polarity= 1
    > module.num[0].m2l.rx.lane[1].polarity= 1
    > module.num[0].m2l.rx.lane[1].local_lane= 1
    > module.num[0].m2l.tx.lane[1].local_lane= 1
    > module.num[0].m2l.rx.lane[1].local_port= 0
    > module.num[0].m2l.tx.lane[1].local_port= 0
    > module.num[0].m2l.tx.lane[2].serdes= 1
    > module.num[0].m2l.rx.lane[2].serdes= 2
    > module.num[0].m2l.tx.lane[2].polarity= 0
    > module.num[0].m2l.rx.lane[2].polarity= 0
    > module.num[0].m2l.rx.lane[2].local_lane= 2
    > module.num[0].m2l.tx.lane[2].local_lane= 2
    > module.num[0].m2l.rx.lane[2].local_port= 0
    > module.num[0].m2l.tx.lane[2].local_port= 0
    > module.num[0].m2l.tx.lane[3].serdes= 0
    > module.num[0].m2l.rx.lane[3].serdes= 3
    > module.num[0].m2l.tx.lane[3].polarity= 1
    > module.num[0].m2l.rx.lane[3].polarity= 1
    > module.num[0].m2l.rx.lane[3].local_lane= 3
    > module.num[0].m2l.tx.lane[3].local_lane= 3
    > module.num[0].m2l.rx.lane[3].local_port= 0
    > module.num[0].m2l.tx.lane[3].local_port= 0
    > module.num[0].smbus_slv_indx= 1
    > module.num[0].module_power_level_supported= 6
    >
    
     
    #9
  10. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,469
    Likes Received:
    503
    Thanks, will have to read up how to do that first but will give it a try:)
     
    #10

Share This Page