10gbit interface compatibility (Intel X553 <-> Mellanox ConnectX-2)

microlith

New Member
Aug 12, 2018
7
0
1
I've been working on a Supermicro SYS-E300-9A (Intel Atom C3858,) putting together a fully virtualized router + whatever, and I'm getting around to being complete. The last step for me is routing my existing 10gbit setup through it, but I've hit a snag.

My NAS has a Mellanox ConnectX-2 card that previously ran into an identical card on my desktop using a direct attach cable. I am trying to route it into the Supermicro, but I have not yet been able to get it to link up.

Some notes:
  • Initial interconnect was a Cisco direct attach cable. Worked fine between Mellanox adapters, but would not link up between the X553s in the Supermicro, let alone between Mellanox and Intel.
  • New direct attach cable worked between like adapters, but not between Mellanox and Intel.
  • Picked up some Finisar transceivers and one Intel transceiver (to be safe) and between like adapters, everything works with any permutation of transceivers. Between Mellanox and Intel, nothing works.
  • Mellanox-delivered mlnx_en driver ignores ConnectX-2.
  • Intel out-of-tree drivers have been tried, no change.
I'm at a bit at a loss with regards to these adapters, if anyone has some further troubleshooting ideas I am all ears.
 

i386

Well-Known Member
Mar 18, 2016
1,936
505
113
31
Germany
Initial interconnect was a Cisco direct attach cable. Worked fine between Mellanox adapters, but would not link up between the X553s in the Supermicro, let alone between Mellanox and Intel.
I have cisco 3m dacs (different generations) that work fine with cx2/3 and intel nics in xeon d boards oO
Mellanox-delivered mlnx_en driver ignores ConnectX-2.
The cx2 nics use the mlnx4 driver like cx3, you should be able to use these drivers for the cx2.

Firmware up to date?
Drivers up to date?
What os (other than freenas) are you using?
 
  • Like
Reactions: microlith

microlith

New Member
Aug 12, 2018
7
0
1
Firmware is up to date per Mellanox's installer package (2.9.1000), but there's a newer 2.9.1200 revision on the Mellanox site and I'm trying to get that installed manually since their tool won't do it.

OS is Ubuntu 16.04 server.

Now that I test with mstconfig apparently mellanox has dropped support in their delivered drivers and their tools. I was able to update to 2.9.1200 without issue, but it looks like to use their drivers I'd need to roll back some revisions...

And that appears to have worked. Last driver package with support for this controller has given me a link up. Though it appears to be illusory, as I'm not seeing it on the Intel side.
 
Last edited:

saivert

Member
Nov 2, 2015
122
13
18
36
Norway
As long as you use fiber optics everything should speak 10GBASE-SR and thus work. That is a defined standard. If the NICs don't communicate they use some vendor lock-in tricks and don't follow the standard.

First make sure your NIC is actually working and that the driver recognizes it.

I was using 2.9.1200 with current drivers without a problem when I was using Connect-X2 and was connecting with a Intel X520 without issue. Using a generic fs.com transceiver on the ConnectX-2 and Intel transceiver on the X520 NIC. I never installed a driver as the one included in Linux kernel works fine. Same on Windows side using in-box driver although it is recommended to install the Mellanox provided driver as it has more features and tweaking.

Direct attach cables are another story as you are stuck with the same SFP+ module in both ends and some of these may not be compatible with either NIC.

You may attempt to disable auto negotiation and select 10Gbit speed manually.
 

microlith

New Member
Aug 12, 2018
7
0
1
Neither NIC supports autonegotiation, with the only difference I've seen in ethtool being the use of pause frames.

I have an X520 card on the way, I'll swap out the Mellanox and see if that doesn't resolve things.
 

microlith

New Member
Aug 12, 2018
7
0
1
So after swapping out the Mellanox for the X520 I still am having no luck. The new card establishes a link over fiber with the Mellanox in my desktop without issue, but the X553 adapter refuses to link up with anything other than the other X553 in the same machine.

Guess it's time to contact Supermicro for support, this is ridiculous.
 

compuguy

New Member
Aug 23, 2017
15
4
3
So after swapping out the Mellanox for the X520 I still am having no luck. The new card establishes a link over fiber with the Mellanox in my desktop without issue, but the X553 adapter refuses to link up with anything other than the other X553 in the same machine.

Guess it's time to contact Supermicro for support, this is ridiculous.
Good luck with that. I've been having similar issues with a E300-8D (It uses the same Intel SOC chipset). I contacted Supermicro support, and they've given me mixed information on copper and fiber modules (I don't know about direct attach cables). Supermicro support did mention one DAC they sell on their store, the CBL-NTWK-0347: Supermicro 10G SFP+ Passive Copper 1M Push Type Cable (CBL-NTWK-0347) (I have personally not tried it). The other possibility is to use a short fiber patch cable. They seem to like genuine Cisco SFP-10G-SR's, made by Finisar and Avago (Foxconn). I have a feeling that you may have luck using a SFP+ module made by those two OEM's.

Edit: it seems to also like SFP/SFP+ modules that are Intel or at least have a matching Vendor OUI for intel.
 
Last edited:

microlith

New Member
Aug 12, 2018
7
0
1
Tried the patch cable with Intel and Finisar units. One of the Intel units is a branded Finisar module.

I am already in contact with Supermicro, and I'm pushing them to check the proper SFP+ units and DACs.
 

compuguy

New Member
Aug 23, 2017
15
4
3
Tried the patch cable with Intel and Finisar units. One of the Intel units is a branded Finisar module.

I am already in contact with Supermicro, and I'm pushing them to check the proper SFP+ units and DACs.
Let me know what happens with that. They've been a mixed bag on compatible modules (especially if you want a RJ45 one). They told me the Aquantia 10G/s High Speed SFP+ Ethernet Optical Transceiver should be compatible. I'm not 100% sure to believe them if the previously recommended patch cable doesn't work.
 

microlith

New Member
Aug 12, 2018
7
0
1
Ha wow, the irony with that unit is the system I have has two of those sockets directly adjacent to the SFP+ ports, and my desktop has an Aquantia controller built in, which was half the reason for this system.
 

Vaka

New Member
Oct 28, 2018
1
0
1
I had such problem too with Supermicro (intel x553) adapter.
I found that kernel module 'ixgbe' have parameter: allow_unsupported_sfp.
So...
# rmmod ixgbe
# echo "options ixgbe allow_unsupported_sfp=1" > /etc/modprobe.d/ixgbe.conf
# modprobe ixgbe

Now connection works with my Cisco DAC