Anybody w/ Mellanox Support contract ?

Rand__

Well-Known Member
Mar 6, 2014
6,128
1,495
113
Looking for somebody with a Mellanox Support contract whi would be willing to help out with an incompatibility issue ;)

Not even usre it would work (no idea of their support model), but asking cant hurt;)

Long story:
Got a Celestica 100G switch (https://forums.servethehome.com/index.php?threads/100-gbps-ethernet-switch-1000-new.22994) and want to run it with Mellanox CX4's. Unfortunately they dont link up.
After months goinf back and forth on their community page
https://community.mellanox.com/s/qu...was-interoperability-of-cx4-with-sonic-switch
they have no given up and say its a compatibility issue with the switch.
They are not willing to investigatre further without a support contract. I inquired re cost but am fairly sure I will not be willing to pay for that :/

So looking for alternatives now;)
 

Rand__

Well-Known Member
Mar 6, 2014
6,128
1,495
113
I have been going back and forth with mellanox for months - do you think i checked port type ?;)
Yes i did :)
The CX4 links up at 40G but not at 100G, CX5's link up at 40 & 100.
If I had the cash I'd switch to cx5's and be done with it
 

Rand__

Well-Known Member
Mar 6, 2014
6,128
1,495
113
So it does not seem likely i can get support from Mellanox. No warranty for used hardware, the only non time based option is a 5 case package for 5 grands+ (expiring after 12 months to boot).

Last chance is the option to buy a new card via official retailer but no feedback yet whether thats applicable to get the kind of support i'd require:/
 

nephri

Active Member
Sep 23, 2015
540
107
43
44
Paris, France
100Gb switch.... it's a monster...

Connectx-4 are not cheap but still reasonnable...
But connectx-5 are out of reasonnable costs

But the celestica seems to be a very good switch.
 

Rand__

Well-Known Member
Mar 6, 2014
6,128
1,495
113
Yeah would be if I could use it;)

Its annoying that Mellanox is trying to charge me to provide compatibility, but o/c its a competitors product (broadcom based) , so understandable on the other hand. Tried the third time today to get a response from Celestica, but have little hope.

Might need to look for other 100G cards or bury the idea (or wait 5 yrs until the CX5 become affordable)
 

oddball

Active Member
May 18, 2018
204
97
28
41
I had issues like this with a Mellanox card and an Arista switch. I had two cards, both same firmware, one worked and one didn't. I could swap cables until I was blue in the face and nothing changed.

I'm convinced that it was the card. The switch never had an issue with any other cards (Cisco or Mellanox) and all of my cables continued to work.

For what it's worth it was a DAC. I've come to learn that if you want 100% compatibility you need to use branded (or faked) transceivers on either end.

Before I gave up I was far down the rabbit hole looking at specialty branded Mellanox cables that were expensive. It might have been a Connectx-4 that I had issues with as well. I've had Connectx-2 and Connectx-3's work without issue.

Is the card a branded one by chance? Mine was branded that I flashed to generic, but I think the issue was lurking in there. From that point I only purchased non-branded Mellanox and never had an issue again.
 

Rand__

Well-Known Member
Mar 6, 2014
6,128
1,495
113
Just wanted to update this thread (and the others) -
I manged to get CX4's running with the Celestica switch by adjusting FEC auto-negotiation to RS instead of FC or none. With that setting the CX4's link up just fine with any brand of cable I tried.
 

Rand__

Well-Known Member
Mar 6, 2014
6,128
1,495
113
Yeah, given that I now have like 10 100G Cards of various brands and 3 to 4 kinds of cables (not to mention 3 100G switches) it sure is...

O/c I have been drive testing in between only to find that I probably don't need 100G in the first place;)
(To be precise I never expected to fully utilize 100G o/c but its main advantage was the increase of 10/14 GBit to 25 GBit single channel performance. Unfortunately it seems that ZFS is not scaling too well with more drives on few threads).

So at this point the main advantage might be for ECN support which the old FDR MLX switches don't support.
Ah well, it sure makes for fun testing.
Did some perf tests yesterday for the FreeNas forum (ppl doubting 100G capabilities of Chelsio 100G Nics), managed to get 54GBit with almost no tweaking (and some 4GB/s SMB3 performance with QD16/J16) which sure is nice:)
 
Last edited:

Nate Carlson

New Member
Aug 27, 2019
13
6
3
Thanks for the pointer in my thread, where I failed to find this thread.. for what it's worth, I had to set FEC to RS on the Celestica side; changing the card side didn't do it. On my switch running SONiC, the following does it:
Code:
sudo config interface fec Ethernet60 rs
 
  • Like
Reactions: RageBone

Nate Carlson

New Member
Aug 27, 2019
13
6
3
Looks like there is already a PR in SONiC to make RS the default on 100gbit ports.. hopefully it'll get pulled in eventually, it'd help! Certainly doesn't hurt to understand this stuff yourself too though.

 
  • Like
Reactions: im.thatoneguy