Mellanox ConnectX 3 VPI (MCX354A-FCBT) HP OEM but with Mellanox OEM Firmware - 40 USD each

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43

mpeter

New Member
Feb 15, 2019
3
4
3
Bricked other devices in the past, its worth $10 to me to not have to learn or deal with this specific procedure. Might be simple, might not. Don’t care, still worth $10 to have it flashed by someone else.
 
  • Like
Reactions: Sleyk

herby

Active Member
Aug 18, 2013
187
53
28
Mellanox noob here... Will these work in non hp systems?
Yep, I have a couple in Supermicro boards now that I flashed myself with instructions found on these forums. I did have to do the tape over the B-5/B-6 pins trick on one NIC because it had an SMBus conflict with the motherboard, but that's not much of a hastle.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
Yep, I have a couple in Supermicro boards now that I flashed myself with instructions found on these forums. I did have to do the tape over the B-5/B-6 pins trick on one NIC because it had an SMBus conflict with the motherboard, but that's not much of a hastle.
I had to do this for one machine too, I had no problem in about 10 other systems.
 

jpk

Member
Nov 6, 2015
66
29
18
45
I don't know if it's appropriate to post this here, so I can delete if necessary, but here are the instructions I wrote down for flashing the MCX354A cards I bought. I have used mine in Supermicro, Dell and Oracle servers.

For CentOS 7

download firmware tools from:
http://www.mellanox.com/page/management_tools

which leads you to:
http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-rpm.tgz


start the mellanox stuff with mst and look at what cards are installed:
# mst start
# ls -l /dev/mst
mt4099_pciconf0 mt4099_pci_cr0 mt4099_pci_msix0


Next, download firmware from
http://www.mellanox.com/page/firmware_table_ConnectX3IB

Need the firmware for the MCX354A-FCBT which should has the PSID MT_1090120019, and then extract it to get the .bin
http://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip

then run this for each card, changing the /dev/mst/mt4099_pci_crX number from /dev/mst/

# flint -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin -allow_psid_change burn

This should output something like this:

Current FW version on flash: 2.10.2280
New FW version: 2.42.5000


You are about to replace current PSID on flash - "HP_0280210019" with a different PSID - "MT_1090120019".
Note: It is highly recommended not to change the PSID.

Do you want to continue ? (y/n) [n] : y
Burning FS2 FW image without signatures - OK
Restoring signature - OK

And now the mlxfwmanager should show something like this:

# mlxfwmanager
Querying Mellanox devices firmware ...

Device #1:
----------

Device Type: ConnectX3
Part Number: 649281-B21_B1
Description: HP IB 4X FDR CX-3 PCI-e G3 Dual Port HCA
PSID: HP_0280210019
PCI Device Name: /dev/mst/mt4099_pci_cr1
Port1 MAC: 24be059e3f31
Port2 MAC: 24be059e3f32
Versions: Current Available
FW 2.10.2280 N/A
PXE 3.3.0650 N/A

Status: No matching image found

Device #2:
----------

Device Type: ConnectX3
Part Number: MCX354A-FCB_A2-A5
Description: ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
PSID: MT_1090120019
PCI Device Name: /dev/mst/mt4099_pci_cr0
Port1 MAC: 24be059fe0c1
Port2 MAC: 24be059fe0c2
Versions: Current Available
FW 2.42.5000 2.42.5000
FW (Running) 2.10.2280 N/A
PXE 3.4.0752 3.4.0752

Status: Up to date


Change the default port type from VPI (auto) to eth (again, changing 0 to 1, etc for multiple cards)

# mlxconfig -d /dev/mst/mt4099_pciconf0 set LINK_TYPE_P1=2
# mlxconfig -d /dev/mst/mt4099_pciconf0 set LINK_TYPE_P2=2
 

WANg

Well-Known Member
Jun 10, 2018
1,302
967
113
46
New York, NY
Why would you get those for $40 over just flashing the Mlx firmware yourself on the ones for $30 ?
(BO on HP 544QSFP Infiniband 10/40GB FDR/EN PCIe Dual Port HCA 656089-001 649281-B21 | eBay )
It's only a $7 difference unless you buy in bulk and the seller accepts your offer, it's much less risky letting someone else do the firmware swap (regardless of whether it's easy or not it's still your time here), and also, you have a choice between SFF and normal form factor brackets - I am pretty sure that the Mellanox SFF bracket alone is worth $10 including shipping and handling.
 
  • Like
Reactions: Sleyk

101

Member
Apr 30, 2018
35
12
8
Seller is clearly adding value, and they have more units than they know what to do with. Great option, and good on them for seizing the opportunity.
 
  • Like
Reactions: Techspin and Sleyk

vv111y

Member
May 6, 2011
76
4
8
Niagara Falls, Canada
As of today they are still selling. This seems like the best deal for high speed nics.
Just to confirm: this card provides 40gbe for tcp/ip traffic?
Next question is cabling: I need ~10m (32') and this goes through some tight framing so the bulky sfp ends will be a problem. Any recommendations?
Thanks.

ADD: the unflashed option seems to be OOS
ADD2: Any issues with p2p connections using both ports?
 
Last edited:

Fallen Kell

Member
Mar 10, 2020
57
23
8
As of today they are still selling. This seems like the best deal for high speed nics.
Just to confirm: this card provides 40gbe for tcp/ip traffic?
Next question is cabling: I need ~10m (32') and this goes through some tight framing so the bulky sfp ends will be a problem. Any recommendations?
Thanks.

ADD: the unflashed option seems to be OOS
ADD2: Any issues with p2p connections using both ports?
Gonna be tough at that length without going optical. The DACs are great when you are in the same room (or better still, same rack), but once you get beyond a few meters, you really need to start looking at optical based cables (but these might also have issues if you have tight corners you need to go around since there is a minimum bend radius on the cabling).

As for using both ports, just recognize that while some of the ConnectX-3 cards do have dual ports, the ports are more for network redundancy/cable/switch failure usage, not for added bandwidth. The PCI-E 3.0 8x slots that these cards use do not have enough bandwidth to drive both ports at the same time at full bandwidth (I think theoretical there is only ~23 additional Gbps of capacity, not the full 40 needed for the second port, and that is theoretical bandwidth, not practical/real world, so figure only realistically 15-18 additional, this is why the FDR IB network is limited to around 54Gbps).
 
  • Like
Reactions: inf3rno and vv111y