Flashing stock Mellanox firmware to OEM (EMC) ConnectX-3 IB/Ethernet dual-port QSFP adapter

ullbeking

Active Member
Jul 28, 2017
395
33
28
41
London
Hello!

I have the opportunity to purchase some used dual-port Mellanox MCX354A-QCBT ConnectX-3 cards with VPI network adapter. (10 GbE Ethernet and 40 Gbps InfiniBand.) These would be perfect for my cluster after correct configuration..

They were originally pulled from EMC systems and currently have EMC firmware installed. I want to flash the original stock (Mellanox) firmware and return them to functionally equivalant stock Mellanox ConnectX-3 cards.

I found a few threads about this ( https://forums.servethehome.com/index.php?threads/anyone-flashing-mellanox-connectx-3-in-qty.16298/ and https://forums.servethehome.com/ind...lanox-connectx-3-en-with-stock-firmware.8689/ ). These discussions indicate that it's a risky process and that the original firmware files are not easy to find. If it goes horribly wrong for a particular card is it difficult to restore using a hardware flasher?

Has anybody tried this recently, and what have your experiences been? Are there any particular hints or tips that I should follow? I intend to connect them to a Voltaire 4036 or 4036E switch, depending on what's available and how cost-effective the options are. Thanks!!

Kind regards,

ullbeking
 
Last edited:

ullbeking

Active Member
Jul 28, 2017
395
33
28
41
London
i crossflash qcbt to fcbt two month ago ,so far so good
Thank you for the positive report, @Freebsd1976!

Please could you provide a brief outline of what you did, or what instructions you followed, to do this? What host machine and OS did you use? Where did you get the firmware from, and did you do it by software flashing or hardware flashing?
 

Freebsd1976

Active Member
Feb 23, 2018
205
31
28
basically i follow the first link you post
and i use windows flash and software flash
only problem is you need find the firmware link , i found in the this forum but now i cannot remember which post it is
 

ullbeking

Active Member
Jul 28, 2017
395
33
28
41
London
@Freebsd1976, what firmware was on your MCX354A-QCBT before you flashed it to -FCBT? My point is my -QCBT card has EMC firmware and I want to replace it by Mellanox firmware. I think this issue is different from changing -QCBT to -FCBT. Please could you let me know what the firmware was before and after? (OEM or Mellanox.) Thanks!!
 

oddball

Active Member
May 18, 2018
172
57
28
39
I just flashed a card last week as well. If you dig on their website they have all the versions of firmware available for download.

Get their firmware tools zip first. Then interrogate your card and see what you need. Go from there.

The issue I'm facing is the card won't recognize an Arista or Cisco DAC. Might need to break down and buy a Mellanox AOC. I had a similar issue with Cisco cards, they only recognize Cisco cables. My Mellanox branded cards take any DAC I can find.
 

oddball

Active Member
May 18, 2018
172
57
28
39
Oh, you can save off your original firmware and flash back if you make a mistake.
 

ullbeking

Active Member
Jul 28, 2017
395
33
28
41
London
I just flashed a card last week as well. If you dig on their website they have all the versions of firmware available for download.
Yes, I tried digging around but I wasn't sure of the correct one to use. Obviously I want to flash to the latest "good" firmware that makes my card into a standard Mellanox card rather than an EMC OEM card. One potentially major issue on my mind is this: Might there be anything about the EMC OEM hardware or firmware that prevents cross-flashing?

Get their firmware tools zip first. Then interrogate your card and see what you need. Go from there.
The Mellanox firmware tools? Will this work with the EMC firmware? (I haven't actually purchased the card yet but I'm currently submitting an order for express delivery because this is kind of important to me.)

Oh, you can save off your original firmware and flash back if you make a mistake.
Of course. Are there standard flashing tools that I can use to do this? For example, would `flashrom` work?

Please could you give me some examples of useful tools for reading original firmware and flashing new firmware? My understanding from my research so far is that the EMC (or Mellanox?) tools do not allow cross-flashing and that using more flexible or generic tools will be necessary.

Do you know if it's possible to flash the firmware using a hardware flasher? This would obviously require extracting the actual firmware image from the .exe file on the Mellanox website.
 

oddball

Active Member
May 18, 2018
172
57
28
39
You're overthinking this. Mellanox makes it REALLY easy.

They have firmware tools like flint. Look at the first link. Those are all commands from Mellanox's tools. The firmware tools are a download on their site.

You run one of their tools to find the current firmware and PSID. From that you then use the chooser on their website to find the latest that's compatible with your card. You run flint with the new bin file and about 30s later you have new firmware. You can run all of this if the card is physically present but the OS doesn't detect it. They're built to handle undetected cards.

You don't need hardware flashing tools or anything. Their tools allow you to pull off all of the current firmware and save it. Their own tools have everything you'd need. They have really nice documentation that walks you through it as well.

They even have a section of their website with vendor specific details. I re-flashed a Fujitsu card to be a standard Mellanox one.

From what I understand the cards are 100% generic with vendor add-ons in flash only. If this is an EMC version of a Connectx-3 then you're fine.

And just to clarify, you have an EMC version of a Mellanox Connectx-3 that you want to flash to a Mellanox generic Connectx-3 right? Your last post made me think you're trying to toss custom firmware on there. Flint will allow that as well, but that's way out of my league.
 

ullbeking

Active Member
Jul 28, 2017
395
33
28
41
London
Thank you @oddball !! This has been a huge help. I spent the evening experimenting with some of the flashing tools, compiling them, etc, and am getting a better feel now for how this is supposed to work. I’ll let you know how I get on once I receive my experimental HCAs.
 

Hindsight

Member
Mar 28, 2016
55
13
8
38
Hey guys. So before I actually burn my card I want to verify this should be correct. Here is prior info

I bought some MCX354A-QCBT

Image type: FS2
FW Version: 2.30.8020
MIC Version: 1.5.0
Config Sectors: 2
Rom Info: type=PXE version=3.4.151 proto=VPI
Device ID: 4099
Description: Node Port1 Port2 Sys image

VSD:
PSID: ISL1090110018
During the firmware creation process instead of my normal card I choose MCX354A-FCB_A2-A5.ini

This is the firmware info

Image type: FS2
FW Version: 2.40.5000
FW Release Date: 27.10.2016
MIC Version: 2.0.0
Config Sectors: 1
PRS Name: cx3-1_MCX354A_fdr_09v.prs
Device ID: 4099
Description: Node Port1 Port2 Sys image

VSD: n/a
PSID: MT_1090120019
So if thats good I should be all set!
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,906
1,713
113
29
fohdeesha.com
If you're just flashing a MCX354A-QCBT (649281-B21 on ebay) to a MCX354A-FCBT (full 40gbE enabled), you don't need to compile a firmware image - you can just flash the premade latest MCX354A-FCBT image from their site - this is nice because the latest firmware sources available for compiling are actually quite old.

When you use the firmware sources with the stock unedited FCBT ini file to compile a new image, this is all you're generating anyway. Except the pre-generated images on mellanox's site have been compiled with much newer firmware sources

from my notes:

Code:
##Download latest Mellanox Firmware tools and install them  + dependencies:

wget http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-deb.tgz
tar -xvf mft-4.10.0-104-x86_64-deb.tgz
cd mft-4.10.0-104-x86_64-deb/
apt-get install gcc make dkms unzip
apt install linux-headers-$(uname -r)
./install.sh

#Start MST service:
mst start
mst status
#copy the dev address with cr0 in it, like:
/dev/mst/mt4099_pci_cr0
#Use that address in the following commands

#Download latest FCBT firmware and unzip:
wget http://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip
unzip fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip

#Backup the cards current board definition file just in case
flint -d /dev/mst/mt4099_pci_cr0 dc orig_firmware.ini

#Flash the FCBT image to the card
flint -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin -allow_psid_change burn

Cold boot the server when done
Some useful config commands to config the card when it comes back up:

Code:
#start mst service again
mst start

#get detailed firmware info
mlxfwmanager --query

#get the device ID again:
mst status

#use it to see the cards current configuration:
mlxconfig -d /dev/mst/mt4099_pci_cr0 query

use it in config commands to configure the card
#for instance, to turn both ports from VPI/Auto to Ethernet only:
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2


#turn off bootrom crap
mlxconfig -d /dev/mst/mt4099_pci_cr0 set BOOT_OPTION_ROM_EN_P1=false
mlxconfig -d /dev/mst/mt4099_pci_cr0 set BOOT_OPTION_ROM_EN_P2=false
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LEGACY_BOOT_PROTOCOL_P1=0
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LEGACY_BOOT_PROTOCOL_P2=0


##optional: delete bootrom off the card, so it doesn't slow down boot by popping up crap
##this is safe to do and supported by mellanox
flint -d /dev/mst/mt4099_pci_cr0 --allow_rom_change drom
 
Last edited:

Hindsight

Member
Mar 28, 2016
55
13
8
38
If you're just flashing a MCX354A-QCBT to a MCX354A-FCBT, you don't need to compile a firmware image - you can just flash the premade latest MCX354A-FCBT image from their site - this is nice because the latest firmware sources available for compiling are actually quite old.

When you use the firmware sources with the stock unedited FCBT ini file to compile a new image, this is all you're generating anyway. Except the pre-generated images on mellanox's site have been compiled with much newer firmware sources
Ohh great, that was a lot easier. Looks like it worked well. 2 more cards to flash. Thank you!
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
Is it possible to flash to ConnectX-3 Pro MCX354A-FCCT?
I'm interested in this as well -- Are the CX3 Pro cards different hardware or are the features just firmware locked?

Edit: I have two QCBT cards here now, I'm waiting on full height brackets and then I'll try flashing one to FCCT to test.
 
Last edited:
  • Like
Reactions: Hindsight

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
233
43
So no-go on the FCCT firmware, the firmware tools stop you from flashing the Pro firmware on a non Pro card.

Does anyone know if the IC on the pro is actually different or are they just differentiating SKUs to segment the market? If the hardware is identical I'd be happy to crack the firmware tool and try flashing Pro firmware on a non pro.

Also, I can't get my workstation to post with a CX-3 installed. This is the same machine I've had a CX-2 in for the last year, on the off chance this has happened to someone else is there any obvious reason a CX-3 would cause a Z97 board to hang on boot? I'm going to try again this afternoon after I swap cards around and update the firmware and disable PXE boot.