10gbit interface compatibility (Intel X553 <-> Mellanox ConnectX-2)

Discussion in 'Networking' started by microlith, Aug 12, 2018.

  1. microlith

    microlith New Member

    Joined:
    Aug 12, 2018
    Messages:
    6
    Likes Received:
    0
    I've been working on a Supermicro SYS-E300-9A (Intel Atom C3858,) putting together a fully virtualized router + whatever, and I'm getting around to being complete. The last step for me is routing my existing 10gbit setup through it, but I've hit a snag.

    My NAS has a Mellanox ConnectX-2 card that previously ran into an identical card on my desktop using a direct attach cable. I am trying to route it into the Supermicro, but I have not yet been able to get it to link up.

    Some notes:
    • Initial interconnect was a Cisco direct attach cable. Worked fine between Mellanox adapters, but would not link up between the X553s in the Supermicro, let alone between Mellanox and Intel.
    • New direct attach cable worked between like adapters, but not between Mellanox and Intel.
    • Picked up some Finisar transceivers and one Intel transceiver (to be safe) and between like adapters, everything works with any permutation of transceivers. Between Mellanox and Intel, nothing works.
    • Mellanox-delivered mlnx_en driver ignores ConnectX-2.
    • Intel out-of-tree drivers have been tried, no change.
    I'm at a bit at a loss with regards to these adapters, if anyone has some further troubleshooting ideas I am all ears.
     
    #1
  2. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,450
    Likes Received:
    331
    I have cisco 3m dacs (different generations) that work fine with cx2/3 and intel nics in xeon d boards oO
    The cx2 nics use the mlnx4 driver like cx3, you should be able to use these drivers for the cx2.

    Firmware up to date?
    Drivers up to date?
    What os (other than freenas) are you using?
     
    #2
    microlith likes this.
  3. microlith

    microlith New Member

    Joined:
    Aug 12, 2018
    Messages:
    6
    Likes Received:
    0
    Firmware is up to date per Mellanox's installer package (2.9.1000), but there's a newer 2.9.1200 revision on the Mellanox site and I'm trying to get that installed manually since their tool won't do it.

    OS is Ubuntu 16.04 server.

    Now that I test with mstconfig apparently mellanox has dropped support in their delivered drivers and their tools. I was able to update to 2.9.1200 without issue, but it looks like to use their drivers I'd need to roll back some revisions...

    And that appears to have worked. Last driver package with support for this controller has given me a link up. Though it appears to be illusory, as I'm not seeing it on the Intel side.
     
    #3
    Last edited: Aug 13, 2018
  4. saivert

    saivert Member

    Joined:
    Nov 2, 2015
    Messages:
    109
    Likes Received:
    10
    As long as you use fiber optics everything should speak 10GBASE-SR and thus work. That is a defined standard. If the NICs don't communicate they use some vendor lock-in tricks and don't follow the standard.

    First make sure your NIC is actually working and that the driver recognizes it.

    I was using 2.9.1200 with current drivers without a problem when I was using Connect-X2 and was connecting with a Intel X520 without issue. Using a generic fs.com transceiver on the ConnectX-2 and Intel transceiver on the X520 NIC. I never installed a driver as the one included in Linux kernel works fine. Same on Windows side using in-box driver although it is recommended to install the Mellanox provided driver as it has more features and tweaking.

    Direct attach cables are another story as you are stuck with the same SFP+ module in both ends and some of these may not be compatible with either NIC.

    You may attempt to disable auto negotiation and select 10Gbit speed manually.
     
    #4
  5. microlith

    microlith New Member

    Joined:
    Aug 12, 2018
    Messages:
    6
    Likes Received:
    0
    Neither NIC supports autonegotiation, with the only difference I've seen in ethtool being the use of pause frames.

    I have an X520 card on the way, I'll swap out the Mellanox and see if that doesn't resolve things.
     
    #5
  6. microlith

    microlith New Member

    Joined:
    Aug 12, 2018
    Messages:
    6
    Likes Received:
    0
    So after swapping out the Mellanox for the X520 I still am having no luck. The new card establishes a link over fiber with the Mellanox in my desktop without issue, but the X553 adapter refuses to link up with anything other than the other X553 in the same machine.

    Guess it's time to contact Supermicro for support, this is ridiculous.
     
    #6
  7. compuguy

    compuguy New Member

    Joined:
    Aug 23, 2017
    Messages:
    15
    Likes Received:
    4
    Good luck with that. I've been having similar issues with a E300-8D (It uses the same Intel SOC chipset). I contacted Supermicro support, and they've given me mixed information on copper and fiber modules (I don't know about direct attach cables). Supermicro support did mention one DAC they sell on their store, the CBL-NTWK-0347: Supermicro 10G SFP+ Passive Copper 1M Push Type Cable (CBL-NTWK-0347) (I have personally not tried it). The other possibility is to use a short fiber patch cable. They seem to like genuine Cisco SFP-10G-SR's, made by Finisar and Avago (Foxconn). I have a feeling that you may have luck using a SFP+ module made by those two OEM's.

    Edit: it seems to also like SFP/SFP+ modules that are Intel or at least have a matching Vendor OUI for intel.
     
    #7
    Last edited: Aug 27, 2018
  8. microlith

    microlith New Member

    Joined:
    Aug 12, 2018
    Messages:
    6
    Likes Received:
    0
    Tried the patch cable with Intel and Finisar units. One of the Intel units is a branded Finisar module.

    I am already in contact with Supermicro, and I'm pushing them to check the proper SFP+ units and DACs.
     
    #8
  9. compuguy

    compuguy New Member

    Joined:
    Aug 23, 2017
    Messages:
    15
    Likes Received:
    4
    Let me know what happens with that. They've been a mixed bag on compatible modules (especially if you want a RJ45 one). They told me the Aquantia 10G/s High Speed SFP+ Ethernet Optical Transceiver should be compatible. I'm not 100% sure to believe them if the previously recommended patch cable doesn't work.
     
    #9
  10. microlith

    microlith New Member

    Joined:
    Aug 12, 2018
    Messages:
    6
    Likes Received:
    0
    Ha wow, the irony with that unit is the system I have has two of those sockets directly adjacent to the SFP+ ports, and my desktop has an Aquantia controller built in, which was half the reason for this system.
     
    #10
  11. Vaka

    Vaka New Member

    Joined:
    Oct 28, 2018
    Messages:
    1
    Likes Received:
    0
    I had such problem too with Supermicro (intel x553) adapter.
    I found that kernel module 'ixgbe' have parameter: allow_unsupported_sfp.
    So...
    # rmmod ixgbe
    # echo "options ixgbe allow_unsupported_sfp=1" > /etc/modprobe.d/ixgbe.conf
    # modprobe ixgbe

    Now connection works with my Cisco DAC
     
    #11
Similar Threads: 10gbit interface
Forum Title Date
Networking What is a normal amount of Retr in a 10gbit iperf3 test? Nov 27, 2018
Networking Can't get 10Gbit speeds on Intel X550-T1 Jul 3, 2018
Networking Cheap 10Gbit TOR Switch, ES-16-XG? Apr 10, 2017
Networking 10gbit fiber with wall plate run question Apr 16, 2016
Networking Do Mellanox x2/x3 10gbit sfp cards work with Sophos? Mar 21, 2016

Share This Page