Getting a 100gbe link between Celstica DX010 and Mellanox ConnectX-4?

Nate Carlson

New Member
Aug 27, 2019
13
6
3
Howdy!

I've got a Celestica SeaStone DX010 running SONiC, which is working well with a couple QSFP+ 40gbe AOCs and ConnectX-3 adapters. I've picked up a few ConnectX-4 adapters (HP 840QSFP28, CX455A) and three different brands of DACs - but I can't get a link to come up at 100gbe. If I hard-code the switch side to 40gbe, it comes up with the CX4 card and a QSFP28 DAC, but if I try to do 100gbe, the link goes down and I don't see any indication of why it won't come up.

I've flashed the current Mellanox stock firmware to the card without issues, but still won't come up at 100gbe.

If I plug the DAC into two ports on the switch, they loop up at 100gbe. If I connect two NICs together with the DAC, they also both come up at 100gbe. It's just the NIC-to-switch connection that won't come up.

Any tips/tricks?

Thanks!
 

Nate Carlson

New Member
Aug 27, 2019
13
6
3
Your google foo is lacking - i think I have 3 posts here + one on the Mellanox Forum with the solution ;)

_hangs head in shame_

Thanks for that! Apparently my search terms were, indeed, lacking. :)

For future reference, on the Celestica+SONiC side:
Code:
admin@celestica:~$ sudo config interface fec Ethernet60 rs
Modifying this on the Linux side didn't seem to make any difference.

This fixes it for two of the three types of QSFP28 DACs I have - works good with 10Gtek Mellanox-coded DACs from Amazon and HP-branded Intel OmniPath cables, but still fails with the cheapest ones (I think they were $8/ea on eBay, and have no identification label); they will just plain not work with the updated firmware on the CX4 cards; when I plug them in it blinks orange and throws:
Code:
[503991.336566] mlx5_core 0000:02:00.0: port_module:245:(pid 0): Port module event[error]: module 0, Cable error, Bus stuck (I2C or data shorted)
In any case - sorry for being a dunce with my searchfu, and appreciate the pointer! :)
 
  • Like
Reactions: Rand__