Another ConnectX-3 cross-flashing question

Discussion in 'Networking' started by BeTeP, Sep 26, 2019.

  1. BeTeP

    BeTeP Active Member

    Joined:
    Mar 23, 2019
    Messages:
    228
    Likes Received:
    84
    I did not have much experience with high speed networks beyond 10Gbps Ethernet and I was under the impression that any ConnectX-3 with QSFP ports was at least 40GbE capable. But after I finally got myself a 40/56GbE switch (Mellanox SX6012) I quickly found that it was not the case. A few of my systems that I deemed to be 40GbE ready (by mere fact of having QSFP ports) in fact were 10GbE only.

    In this particular system the ConnectX-3 adapter is on-board and the only PCIe slot is taken by an HBA - so the NIC can't be easily upgraded by replacing the card. So before I replace the whole system (which I would have preferred to keep otherwise) I would like to confirm that there is no way that card can be upgraded to 40GbE.

    The exact chip part number is MT27518A1-FCCR-XE. Mellanox used this chip revision in their 10GbE CX3 cards with SFP+ ports and then they switched to MT27518A1-FCCR-BE for their 40GbE adapters.

    Anyone knows the difference between -BE and -XE parts? And is there any chance that -XE can be flashed to support 40GbE?
     
    #1
  2. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    97
    Likes Received:
    7
    why not try flash 40gb firmware and try?
     
    #2
  3. BeTeP

    BeTeP Active Member

    Joined:
    Mar 23, 2019
    Messages:
    228
    Likes Received:
    84
    Because the original card's ini is way different from anything I've seen before. There is no way a card would work with wrong GPIO settings or SerDes mapping for example. I checked all the INI files included with ConnectX3-rel-2_40_5030.tgz - there was nothing even close.
     
    #3
  4. BeTeP

    BeTeP Active Member

    Joined:
    Mar 23, 2019
    Messages:
    228
    Likes Received:
    84
    I am thinking about trying to swap the chip on the board with a chip from a cheap CX354A donor. But I wanted to confirm the pin compatibility before I risk ruining both boards. I can't seem to find any information though.
     
    #4
  5. BeTeP

    BeTeP Active Member

    Joined:
    Mar 23, 2019
    Messages:
    228
    Likes Received:
    84
    After a lot of trial and error (and looking at many ini files for different CX3 LOM cards) I put together a config allowing this MT27518A1-FCCR-XE to link at 40Gbps without apparent errors in the ethtool output. The actual throughput reaches about 25Gbps and I have not identified the bottleneck yet. But this performance is just enough for me to keep this system for now.
     
    #5
  6. Freebsd1976

    Freebsd1976 Member

    Joined:
    Feb 23, 2018
    Messages:
    97
    Likes Received:
    7
    I also have cx3 qcbt , if set it to 40gb only ony port work, if set 10g both port work , if flash it to fcbt on 10g both port work , set 40gb none port work , tried both mellanox or self compiled firmware ,same result
     
    #6
    Last edited: Oct 29, 2019
Similar Threads: Another ConnectX-3
Forum Title Date
Networking Yet another PfSense Question - Building a box ( quiet/1u/passive'ish ) Jul 13, 2018
Networking Infiniband PCIe card preventing boot in one server but not another May 12, 2018
Networking Another Intel Fake? Jul 21, 2017
Networking HP 1810G-24. Questions about adding another switch Mar 5, 2017
Networking Sorry another 10gb setup question Oct 26, 2016

Share This Page