Another ConnectX-3 cross-flashing question

[BeTeP]

I did not have much experience with high speed networks beyond 10Gbps Ethernet and I was under the impression that any ConnectX-3 with QSFP ports was at least 40GbE capable. But after I finally got myself a 40/56GbE switch (Mellanox SX6012) I quickly found that it was not the case. A few of my systems that I deemed to be 40GbE ready (by mere fact of having QSFP ports) in fact were 10GbE only.

In this particular system the ConnectX-3 adapter is on-board and the only PCIe slot is taken by an HBA - so the NIC can't be easily upgraded by replacing the card. So before I replace the whole system (which I would have preferred to keep otherwise) I would like to confirm that there is no way that card can be upgraded to 40GbE.

The exact chip part number is MT27518A1-FCCR-XE. Mellanox used this chip revision in their 10GbE CX3 cards with SFP+ ports and then they switched to MT27518A1-FCCR-BE for their 40GbE adapters.

Anyone knows the difference between -BE and -XE parts? And is there any chance that -XE can be flashed to support 40GbE?

 

[Freebsd1976]

why not try flash 40gb firmware and try？

 

[BeTeP]

Because the original card's ini is way different from anything I've seen before. There is no way a card would work with wrong GPIO settings or SerDes mapping for example. I checked all the INI files included with ConnectX3-rel-2_40_5030.tgz - there was nothing even close.

 

[BeTeP]

I am thinking about trying to swap the chip on the board with a chip from a cheap CX354A donor. But I wanted to confirm the pin compatibility before I risk ruining both boards. I can't seem to find any information though.

 

[BeTeP]

After a lot of trial and error (and looking at many ini files for different CX3 LOM cards) I put together a config allowing this MT27518A1-FCCR-XE to link at 40Gbps without apparent errors in the ethtool output. The actual throughput reaches about 25Gbps and I have not identified the bottleneck yet. But this performance is just enough for me to keep this system for now.

 

[Freebsd1976]

I also have cx3 qcbt , if set it to 40gb only ony port work, if set 10g both port work , if flash it to fcbt on 10g both port work , set 40gb none port work ， tried both mellanox or self compiled firmware ,same result