Flashing stock Mellanox firmware to OEM (EMC) ConnectX-3 IB/Ethernet dual-port QSFP adapter

inf3rno

New Member
Mar 24, 2021
9
1
3
Just a guess, you may have one card in a system's pci-e slot that's only gen 2 x 4 lanes which would net you a max of 16 Gb/s even though the 40Gb link is active.
Gen2 x4 or gen3 x2 lanes would be a good explanation, it really depends on the motherboard. I had only 6 Gbps with my MCX311A-XCAT and it turned out that my mobo supports only gen2 x2 on the second PCIE x16 slot even though the first slot is gen3. I had to change motherboard to be able to fully use SFP+. I am thinking on QSFP+ now. If we can buy these cheap cards and convert them to real Mellanox, then SFP+ and QSFP+ have almost the same price.
 

Archos

New Member
Apr 6, 2021
3
3
3
Can I use a 40G DAC breakout cable to connect HP 656089-001 Infiniband 10/40GB DP NIC 649281-B21 to a singel SFP+ port on the Brocade ICX 6450 and get a singel 10G link? Seems like thats the cheapest option for optics if it works.
 

inf3rno

New Member
Mar 24, 2021
9
1
3
Can I use a 40G DAC breakout cable to connect HP 656089-001 Infiniband 10/40GB DP NIC 649281-B21 to a singel SFP+ port on the Brocade ICX 6450 and get a singel 10G link? Seems like thats the cheapest option for optics if it works.
You can do it with Mellanox cards and it is cheaper than a QSA.
 

fossxplorer

Active Member
Mar 17, 2016
504
81
28
Oslo, Norway
I have a couple of ConnectX-3 with 2 Proa. 314-BCCT IIRC. These are not HP branded. Are these just locked to ETH? I have 40GbE working nicely, but wondering about IB support.
Are those HP nics visually branded with a sticker on it?
 

Psynapsx

New Member
Feb 21, 2021
29
4
3
I bought HP 764284-B21. I think it's a Mellanox MCX353A-FCCT.
Do I have to flash the stock Mellanox firmware to use it in my Supermicro server?
 

ycp

Member
Jun 22, 2014
182
8
18
Hello I was able to flash a MCX354A-QCBT card to FCBT latest firmware.
The flash finished successfully and i was able to get the 40gbps link working to my Dell s4810 switch.
But when i restart the server it stops working.
In the windows server 2019 device manager i get a yellow exclamation mark.
The ethernet ports stop working.
What could be wrong and what should i do differently?
 

Terry Wallace

PsyOps SysOp
Aug 13, 2018
181
110
43
Central Time Zone
Are you running infiniband / ethernet or VPI mode.
Frequently if your running ethernet and the cards are set to VPI mode (autodetect and run either or) you will have problems when the windows machines start. Look up how to set the PORT_TYPE use mellanox firmware tools. If you set it with the firmware tools it gets written to card and stays in the mode until manually changed.
The VPI mode frequently times out the windows driver before the card has decided what type of switch its talking to.
 
  • Like
Reactions: nedimzukic2

ycp

Member
Jun 22, 2014
182
8
18
@Terry Wallace I have set the PORT_TYPE using the firmware tools to ethernet mode.
The card starts with Ethernet mode by default but the drivers are showing an error (Yellow Exclamation mark in Device Manager)
 

LodeRunner

Active Member
Apr 27, 2019
221
89
28

LodeRunner

Active Member
Apr 27, 2019
221
89
28
Maybe it's different for the CX3 Pros. Here's the driver information on my server:
1637084823105.png

Given that the Information tab is missing in your images, I think your system is not loading the correct driver, unless the MT04099's don't get that driver panel. My workstation has a MCX311A-XCAT and it uses the same version, but adds tabs for Teaming and VLAN settings (but that may also be a Win 10 vs Server 2019 thing).
 
Last edited: