Anybody w/ Mellanox Support contract ?

[Rand__]

Looking for somebody with a Mellanox Support contract whi would be willing to help out with an incompatibility issue

Not even usre it would work (no idea of their support model), but asking cant hurt

Long story:
Got a Celestica 100G switch (https://forums.servethehome.com/index.php?threads/100-gbps-ethernet-switch-1000-new.22994) and want to run it with Mellanox CX4's. Unfortunately they dont link up.
After months goinf back and forth on their community page
https://community.mellanox.com/s/qu...was-interoperability-of-cx4-with-sonic-switch
they have no given up and say its a compatibility issue with the switch.
They are not willing to investigatre further without a support contract. I inquired re cost but am fairly sure I will not be willing to pay for that :/

So looking for alternatives now

 

[i386]

Did you Check the Port Type of your nics?

 

[Rand__]

I have been going back and forth with mellanox for months - do you think i checked port type ?
Yes i did
The CX4 links up at 40G but not at 100G, CX5's link up at 40 & 100.
If I had the cash I'd switch to cx5's and be done with it

 

[Rand__]

So it does not seem likely i can get support from Mellanox. No warranty for used hardware, the only non time based option is a 5 case package for 5 grands+ (expiring after 12 months to boot).

Last chance is the option to buy a new card via official retailer but no feedback yet whether thats applicable to get the kind of support i'd require:/

 

[nephri]

100Gb switch.... it's a monster...

Connectx-4 are not cheap but still reasonnable...
But connectx-5 are out of reasonnable costs

But the celestica seems to be a very good switch.

 

[Rand__]

Yeah would be if I could use it

Its annoying that Mellanox is trying to charge me to provide compatibility, but o/c its a competitors product (broadcom based) , so understandable on the other hand. Tried the third time today to get a response from Celestica, but have little hope.

Might need to look for other 100G cards or bury the idea (or wait 5 yrs until the CX5 become affordable)

 

[oddball]

I had issues like this with a Mellanox card and an Arista switch. I had two cards, both same firmware, one worked and one didn't. I could swap cables until I was blue in the face and nothing changed.

I'm convinced that it was the card. The switch never had an issue with any other cards (Cisco or Mellanox) and all of my cables continued to work.

For what it's worth it was a DAC. I've come to learn that if you want 100% compatibility you need to use branded (or faked) transceivers on either end.

Before I gave up I was far down the rabbit hole looking at specialty branded Mellanox cables that were expensive. It might have been a Connectx-4 that I had issues with as well. I've had Connectx-2 and Connectx-3's work without issue.

Is the card a branded one by chance? Mine was branded that I flashed to generic, but I think the issue was lurking in there. From that point I only purchased non-branded Mellanox and never had an issue again.

 

[oddball]

By branded I mean Dell, HPE, Fujitsu etc.

 

[Rand__]

I think i tried both branded and non branded ones, but thanks

 

[Rand__]

Just wanted to update this thread (and the others) -
I manged to get CX4's running with the Celestica switch by adjusting FEC auto-negotiation to RS instead of FC or none. With that setting the CX4's link up just fine with any brand of cable I tried.

 

[T_Minus]

Well that's awesome, and probably a huge relief for you @Rand__
back to drive testing? lol

 

[Rand__]

Yeah, given that I now have like 10 100G Cards of various brands and 3 to 4 kinds of cables (not to mention 3 100G switches) it sure is...

O/c I have been drive testing in between only to find that I probably don't need 100G in the first place
(To be precise I never expected to fully utilize 100G o/c but its main advantage was the increase of 10/14 GBit to 25 GBit single channel performance. Unfortunately it seems that ZFS is not scaling too well with more drives on few threads).

So at this point the main advantage might be for ECN support which the old FDR MLX switches don't support.
Ah well, it sure makes for fun testing.
Did some perf tests yesterday for the FreeNas forum (ppl doubting 100G capabilities of Chelsio 100G Nics), managed to get 54GBit with almost no tweaking (and some 4GB/s SMB3 performance with QD16/J16) which sure is nice

 

[Nate Carlson]

Thanks for the pointer in my thread, where I failed to find this thread.. for what it's worth, I had to set FEC to RS on the Celestica side; changing the card side didn't do it. On my switch running SONiC, the following does it:

sudo config interface fec Ethernet60 rs

 

[Rand__]

o/c on Celestica side, sorry if that was not clear.
In one of the threads i had the command to use, apparently not here

 

[Nate Carlson]

Looks like there is already a PR in SONiC to make RS the default on 100gbit ports.. hopefully it'll get pulled in eventually, it'd help! Certainly doesn't hurt to understand this stuff yourself too though.

[config_samples.py] 100G port enable fec by default by tengfei-astfs · Pull Request #3775 · sonic-net/sonic-buildimage

Signed-off-by: tengfei tengfei@asterfusion.com - What I did 100G port enable fec by default in t1 and l2 template - How I did it If a port's speed is 100G , the enable fec:rs - How to verify it...

github.com