10gbit interface compatibility (Intel X553 <-> Mellanox ConnectX-2)

Discussion in 'Networking' started by microlith, Aug 12, 2018.

  1. microlith

    microlith New Member

    Joined:
    Aug 12, 2018
    Messages:
    4
    Likes Received:
    0
    I've been working on a Supermicro SYS-E300-9A (Intel Atom C3858,) putting together a fully virtualized router + whatever, and I'm getting around to being complete. The last step for me is routing my existing 10gbit setup through it, but I've hit a snag.

    My NAS has a Mellanox ConnectX-2 card that previously ran into an identical card on my desktop using a direct attach cable. I am trying to route it into the Supermicro, but I have not yet been able to get it to link up.

    Some notes:
    • Initial interconnect was a Cisco direct attach cable. Worked fine between Mellanox adapters, but would not link up between the X553s in the Supermicro, let alone between Mellanox and Intel.
    • New direct attach cable worked between like adapters, but not between Mellanox and Intel.
    • Picked up some Finisar transceivers and one Intel transceiver (to be safe) and between like adapters, everything works with any permutation of transceivers. Between Mellanox and Intel, nothing works.
    • Mellanox-delivered mlnx_en driver ignores ConnectX-2.
    • Intel out-of-tree drivers have been tried, no change.
    I'm at a bit at a loss with regards to these adapters, if anyone has some further troubleshooting ideas I am all ears.
     
    #1
  2. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,324
    Likes Received:
    302
    I have cisco 3m dacs (different generations) that work fine with cx2/3 and intel nics in xeon d boards oO
    The cx2 nics use the mlnx4 driver like cx3, you should be able to use these drivers for the cx2.

    Firmware up to date?
    Drivers up to date?
    What os (other than freenas) are you using?
     
    #2
    microlith likes this.
  3. microlith

    microlith New Member

    Joined:
    Aug 12, 2018
    Messages:
    4
    Likes Received:
    0
    Firmware is up to date per Mellanox's installer package (2.9.1000), but there's a newer 2.9.1200 revision on the Mellanox site and I'm trying to get that installed manually since their tool won't do it.

    OS is Ubuntu 16.04 server.

    Now that I test with mstconfig apparently mellanox has dropped support in their delivered drivers and their tools. I was able to update to 2.9.1200 without issue, but it looks like to use their drivers I'd need to roll back some revisions...

    And that appears to have worked. Last driver package with support for this controller has given me a link up. Though it appears to be illusory, as I'm not seeing it on the Intel side.
     
    #3
    Last edited: Aug 13, 2018
  4. saivert

    saivert Member

    Joined:
    Nov 2, 2015
    Messages:
    103
    Likes Received:
    9
    As long as you use fiber optics everything should speak 10GBASE-SR and thus work. That is a defined standard. If the NICs don't communicate they use some vendor lock-in tricks and don't follow the standard.

    First make sure your NIC is actually working and that the driver recognizes it.

    I was using 2.9.1200 with current drivers without a problem when I was using Connect-X2 and was connecting with a Intel X520 without issue. Using a generic fs.com transceiver on the ConnectX-2 and Intel transceiver on the X520 NIC. I never installed a driver as the one included in Linux kernel works fine. Same on Windows side using in-box driver although it is recommended to install the Mellanox provided driver as it has more features and tweaking.

    Direct attach cables are another story as you are stuck with the same SFP+ module in both ends and some of these may not be compatible with either NIC.

    You may attempt to disable auto negotiation and select 10Gbit speed manually.
     
    #4
  5. microlith

    microlith New Member

    Joined:
    Aug 12, 2018
    Messages:
    4
    Likes Received:
    0
    Neither NIC supports autonegotiation, with the only difference I've seen in ethtool being the use of pause frames.

    I have an X520 card on the way, I'll swap out the Mellanox and see if that doesn't resolve things.
     
    #5
  6. microlith

    microlith New Member

    Joined:
    Aug 12, 2018
    Messages:
    4
    Likes Received:
    0
    So after swapping out the Mellanox for the X520 I still am having no luck. The new card establishes a link over fiber with the Mellanox in my desktop without issue, but the X553 adapter refuses to link up with anything other than the other X553 in the same machine.

    Guess it's time to contact Supermicro for support, this is ridiculous.
     
    #6
Similar Threads: 10gbit interface
Forum Title Date
Networking Can't get 10Gbit speeds on Intel X550-T1 Jul 3, 2018
Networking Cheap 10Gbit TOR Switch, ES-16-XG? Apr 10, 2017
Networking 10gbit fiber with wall plate run question Apr 16, 2016
Networking Do Mellanox x2/x3 10gbit sfp cards work with Sophos? Mar 21, 2016
Networking Brocade MP8000 - 10Gbit switches Apr 13, 2015

Share This Page