100 gig ethernet - which non Mellanox Cards?

Rand__

Well-Known Member
Mar 6, 2014
4,490
876
113
Since my Celestica switch is not working with Mellanox CX4's at 100G I am looking for alternative cards to use.

I could o/c switch to CX5's+ but those are still around $500/card which is a bit too much.

So I am looking for cheaper alternatives that will run with ESXi, and ideally is not too picky re DACs since I got a lot of Mellanox branded ones;)

I have seen the T62100-LP-CR - available for $400 (maybe $300 BO) at this time, that could work, but no idea re DAC compatibility.

I never looked further since I used to be happy with the MLX cards...

Edit:
Found this:
Meanderful: 100G Ethernet NICs - Broadcom joins Mellanox, Cavium's QLogic, and Chelsio

Edit2:
Broadcom uses QSFP56 on the Dual Port cards :/
100Gb Ethernet Network Adapters
 
Last edited:

Labs

Member
Mar 21, 2019
46
5
8
Look on eBay UK for - Dell Intel Omni-Path 100Gb QSFP28 at £180.
I remember it was an article on STH about them but I cannot find it now.
I am not sure about compatibility because there is not much info on the net.
It could be that it works only with Intel QSFP28. Another seller in Germany was selling them for around 160 Euro but didn't buy any since I couldn't find too much info about compatibility.

Edit: I found the one in Germany also but they are 200 Euro not 160 as I remembered.
eBay article number: 123255614885
 

Rand__

Well-Known Member
Mar 6, 2014
4,490
876
113
Yeah i am not sure either how compatible Omni-Path is with anything else tbh..
But thanks.
 

Labs

Member
Mar 21, 2019
46
5
8
Yes, that's why I didn't buy any of those cards but might be an alternative solution to your problem or not...
You can also try some AOC cables or QSFP28 with MPO fiber.
I never had issues with QSFP28 and MPO fiber interconnecting different switches/routers but I had issues with DAC cables where the DAC didn't have the capabilities identified correctly and was flapping or dropping to 40Gbe. Forcing to 100Gbe was disconnecting the link completely.
 

Rand__

Well-Known Member
Mar 6, 2014
4,490
876
113
I have tried 100GB IB cable, Ethernet cable, IB Fibre - only dedicated Ethernet fibre is missing since i didnt have any.
The ones I used were Mellanox certified cables, any of the above worked with CX5's just not CX4's and 100G (they worked with 40G)
 

Labs

Member
Mar 21, 2019
46
5
8
Does mlxlink show anything useful?
https://docs.mellanox.com/display/MFTV4120/mlxlink+Utility
Also, diff config dumps from CX4 and CX5. They can’t be far apart, because they use the same 100G PHY.
Omni-path NICs do not support Ethernet mode...
You are right, it seems the omni-path nics are some interconnect cards not supporting IB or ethernet.

About Mellanox ConnectX ICs it seems they are different between card generations. Do you have any link where is saying CX4 and CX5 use the same IC?
 

necr

Member
Dec 27, 2017
49
20
8
120
About Mellanox ConnectX ICs it seems they are different between card generations. Do you have any link where is saying CX4 and CX5 use the same IC?
I never said they used same IC, but they use same ports/SerDes. New 100G PHY is called Shomron.
As I suspected, differences are tiny. @Rand__ I don't have dual port CX4, but I suggest adjusting polarity on lanes 0 and 2 and also setting power cap to 6.
See diff:

Code:
< module.num[0].label_port= 1
< module.num[0].type= 1
< module.num[0].m2l.tx.lane[0].serdes= 3
< module.num[0].m2l.rx.lane[0].serdes= 0
< module.num[0].m2l.tx.lane[0].polarity= 1
< module.num[0].m2l.rx.lane[0].polarity= 0
< module.num[0].m2l.rx.lane[0].local_port= 0
< module.num[0].m2l.tx.lane[0].local_port= 0
< module.num[0].m2l.tx.lane[1].serdes= 2
< module.num[0].m2l.rx.lane[1].serdes= 1
< module.num[0].m2l.tx.lane[1].polarity= 1
< module.num[0].m2l.rx.lane[1].polarity= 1
< module.num[0].m2l.rx.lane[1].local_port= 0
< module.num[0].m2l.tx.lane[1].local_port= 0
< module.num[0].m2l.tx.lane[2].serdes= 1
< module.num[0].m2l.rx.lane[2].serdes= 2
< module.num[0].m2l.tx.lane[2].polarity= 1
< module.num[0].m2l.rx.lane[2].polarity= 0
< module.num[0].m2l.rx.lane[2].local_port= 0
< module.num[0].m2l.tx.lane[2].local_port= 0
< module.num[0].m2l.tx.lane[3].serdes= 0
< module.num[0].m2l.rx.lane[3].serdes= 3
< module.num[0].m2l.tx.lane[3].polarity= 1
< module.num[0].m2l.rx.lane[3].polarity= 1
< module.num[0].m2l.rx.lane[3].local_port= 0
< module.num[0].m2l.tx.lane[3].local_port= 0
< module.num[0].smbus_slv_indx= 4
< module.num[0].module_power_level_supported= 4
---
> module.num[0].label_port= 1
> module.num[0].type= 1
> module.num[0].m2l.tx.lane[0].serdes= 3
> module.num[0].m2l.rx.lane[0].serdes= 0
> module.num[0].m2l.tx.lane[0].polarity= 0
> module.num[0].m2l.rx.lane[0].polarity= 0
> module.num[0].m2l.rx.lane[0].local_lane= 0
> module.num[0].m2l.tx.lane[0].local_lane= 0
> module.num[0].m2l.rx.lane[0].local_port= 0
> module.num[0].m2l.tx.lane[0].local_port= 0
> module.num[0].m2l.tx.lane[1].serdes= 2
> module.num[0].m2l.rx.lane[1].serdes= 1
> module.num[0].m2l.tx.lane[1].polarity= 1
> module.num[0].m2l.rx.lane[1].polarity= 1
> module.num[0].m2l.rx.lane[1].local_lane= 1
> module.num[0].m2l.tx.lane[1].local_lane= 1
> module.num[0].m2l.rx.lane[1].local_port= 0
> module.num[0].m2l.tx.lane[1].local_port= 0
> module.num[0].m2l.tx.lane[2].serdes= 1
> module.num[0].m2l.rx.lane[2].serdes= 2
> module.num[0].m2l.tx.lane[2].polarity= 0
> module.num[0].m2l.rx.lane[2].polarity= 0
> module.num[0].m2l.rx.lane[2].local_lane= 2
> module.num[0].m2l.tx.lane[2].local_lane= 2
> module.num[0].m2l.rx.lane[2].local_port= 0
> module.num[0].m2l.tx.lane[2].local_port= 0
> module.num[0].m2l.tx.lane[3].serdes= 0
> module.num[0].m2l.rx.lane[3].serdes= 3
> module.num[0].m2l.tx.lane[3].polarity= 1
> module.num[0].m2l.rx.lane[3].polarity= 1
> module.num[0].m2l.rx.lane[3].local_lane= 3
> module.num[0].m2l.tx.lane[3].local_lane= 3
> module.num[0].m2l.rx.lane[3].local_port= 0
> module.num[0].m2l.tx.lane[3].local_port= 0
> module.num[0].smbus_slv_indx= 1
> module.num[0].module_power_level_supported= 6
>
 

Rand__

Well-Known Member
Mar 6, 2014
4,490
876
113
Thanks, will have to read up how to do that first but will give it a try:)
 

Rand__

Well-Known Member
Mar 6, 2014
4,490
876
113
Just wanted to update this thread (and the others) -
I manged to get CX4's running with the Celestica switch by adjusting FEC auto-negotiation to RS instead of FC or none. With that setting the CX4's link up just fine with any brand of cable I tried.
 
  • Like
Reactions: Stephan and necr