Flashing stock Mellanox firmware to OEM (EMC) ConnectX-3 IB/Ethernet dual-port QSFP adapter

[ullbeking]

Hello!

I have the opportunity to purchase some used dual-port Mellanox MCX354A-QCBT ConnectX-3 cards with VPI network adapter. (10 GbE Ethernet and 40 Gbps InfiniBand.) These would be perfect for my cluster after correct configuration..

They were originally pulled from EMC systems and currently have EMC firmware installed. I want to flash the original stock (Mellanox) firmware and return them to functionally equivalant stock Mellanox ConnectX-3 cards.

I found a few threads about this ( https://forums.servethehome.com/index.php?threads/anyone-flashing-mellanox-connectx-3-in-qty.16298/ and https://forums.servethehome.com/ind...lanox-connectx-3-en-with-stock-firmware.8689/ ). These discussions indicate that it's a risky process and that the original firmware files are not easy to find. If it goes horribly wrong for a particular card is it difficult to restore using a hardware flasher?

Has anybody tried this recently, and what have your experiences been? Are there any particular hints or tips that I should follow? I intend to connect them to a Voltaire 4036 or 4036E switch, depending on what's available and how cost-effective the options are. Thanks!!

Kind regards,

ullbeking

 

[Freebsd1976]

i crossflash qcbt to fcbt two month ago ,so far so good

 

[ullbeking]

i crossflash qcbt to fcbt two month ago ,so far so good

Thank you for the positive report, @Freebsd1976!

Please could you provide a brief outline of what you did, or what instructions you followed, to do this? What host machine and OS did you use? Where did you get the firmware from, and did you do it by software flashing or hardware flashing?

 

[Freebsd1976]

basically i follow the first link you post
and i use windows flash and software flash
only problem is you need find the firmware link , i found in the this forum but now i cannot remember which post it is

 

[Freebsd1976]

found the post , in the post there are two links about firmware, both 2.40.5000.tgz and 2.40.5030.tgz,and i use the first one.

https://forums.servethehome.com/index.php?threads/connectx-3-hp-to-mellanox-firmware-help.12452/page-2

 

[ullbeking]

found the post , in the post there are two links about firmware, both 2.40.5000.tgz and 2.40.5030.tgz,and i use the first one.

https://forums.servethehome.com/index.php?threads/connectx-3-hp-to-mellanox-firmware-help.12452/page-2

@Freebsd1976, you rock my world. You really do. Thank you.

 

[ullbeking]

@Freebsd1976, what firmware was on your MCX354A-QCBT before you flashed it to -FCBT? My point is my -QCBT card has EMC firmware and I want to replace it by Mellanox firmware. I think this issue is different from changing -QCBT to -FCBT. Please could you let me know what the firmware was before and after? (OEM or Mellanox.) Thanks!!

 

[Freebsd1976]

Mellanox firmware, I think Flash Oem to mellanox should be same

 

[oddball]

I just flashed a card last week as well. If you dig on their website they have all the versions of firmware available for download.

Get their firmware tools zip first. Then interrogate your card and see what you need. Go from there.

The issue I'm facing is the card won't recognize an Arista or Cisco DAC. Might need to break down and buy a Mellanox AOC. I had a similar issue with Cisco cards, they only recognize Cisco cables. My Mellanox branded cards take any DAC I can find.

 

[oddball]

Oh, you can save off your original firmware and flash back if you make a mistake.

 

[ullbeking]

I just flashed a card last week as well. If you dig on their website they have all the versions of firmware available for download.

Yes, I tried digging around but I wasn't sure of the correct one to use. Obviously I want to flash to the latest "good" firmware that makes my card into a standard Mellanox card rather than an EMC OEM card. One potentially major issue on my mind is this: Might there be anything about the EMC OEM hardware or firmware that prevents cross-flashing?

Get their firmware tools zip first. Then interrogate your card and see what you need. Go from there.

The Mellanox firmware tools? Will this work with the EMC firmware? (I haven't actually purchased the card yet but I'm currently submitting an order for express delivery because this is kind of important to me.)

Oh, you can save off your original firmware and flash back if you make a mistake.

Of course. Are there standard flashing tools that I can use to do this? For example, would `flashrom` work?

Please could you give me some examples of useful tools for reading original firmware and flashing new firmware? My understanding from my research so far is that the EMC (or Mellanox?) tools do not allow cross-flashing and that using more flexible or generic tools will be necessary.

Do you know if it's possible to flash the firmware using a hardware flasher? This would obviously require extracting the actual firmware image from the .exe file on the Mellanox website.

 

[oddball]

You're overthinking this. Mellanox makes it REALLY easy.

They have firmware tools like flint. Look at the first link. Those are all commands from Mellanox's tools. The firmware tools are a download on their site.

You run one of their tools to find the current firmware and PSID. From that you then use the chooser on their website to find the latest that's compatible with your card. You run flint with the new bin file and about 30s later you have new firmware. You can run all of this if the card is physically present but the OS doesn't detect it. They're built to handle undetected cards.

You don't need hardware flashing tools or anything. Their tools allow you to pull off all of the current firmware and save it. Their own tools have everything you'd need. They have really nice documentation that walks you through it as well.

They even have a section of their website with vendor specific details. I re-flashed a Fujitsu card to be a standard Mellanox one.

From what I understand the cards are 100% generic with vendor add-ons in flash only. If this is an EMC version of a Connectx-3 then you're fine.

And just to clarify, you have an EMC version of a Mellanox Connectx-3 that you want to flash to a Mellanox generic Connectx-3 right? Your last post made me think you're trying to toss custom firmware on there. Flint will allow that as well, but that's way out of my league.

 

[oddball]

On the second link (https://forums.servethehome.com/ind...lanox-connectx-3-en-with-stock-firmware.8689/)

Post #11 are the exact steps to follow.

 

[ullbeking]

Thank you @oddball !! This has been a huge help. I spent the evening experimenting with some of the flashing tools, compiling them, etc, and am getting a better feel now for how this is supposed to work. I’ll let you know how I get on once I receive my experimental HCAs.

 

[Hindsight]

Hey guys. So before I actually burn my card I want to verify this should be correct. Here is prior info

I bought some MCX354A-QCBT

Image type: FS2
FW Version: 2.30.8020
MIC Version: 1.5.0
Config Sectors: 2
Rom Info: type=PXE version=3.4.151 proto=VPI
Device ID: 4099
Description: Node Port1 Port2 Sys image

VSD:
PSID: ISL1090110018

During the firmware creation process instead of my normal card I choose MCX354A-FCB_A2-A5.ini

This is the firmware info

Image type: FS2
FW Version: 2.40.5000
FW Release Date: 27.10.2016
MIC Version: 2.0.0
Config Sectors: 1
PRS Name: cx3-1_MCX354A_fdr_09v.prs
Device ID: 4099
Description: Node Port1 Port2 Sys image

VSD: n/a
PSID: MT_1090120019

So if thats good I should be all set!

 

[fohdeesha]

If you're just flashing a MCX354A-QCBT (649281-B21 on ebay) to a MCX354A-FCBT (full 40gbE enabled), you don't need to compile a firmware image - you can just flash the premade latest MCX354A-FCBT image from their site - this is nice because the latest firmware sources available for compiling are actually quite old.

When you use the firmware sources with the stock unedited FCBT ini file to compile a new image, this is all you're generating anyway. Except the pre-generated images on mellanox's site have been compiled with much newer firmware sources

from my notes (I usually just do this with a debian or ubuntu live ISO):

##Download latest Mellanox Firmware tools and install them  + dependencies:
apt update && apt install gcc make dkms unzip linux-headers-$(uname -r)
wget https://www.mellanox.com/downloads/MFT/mft-4.18.0-106-x86_64-deb.tgz
tar -xvf mft-4.18.0-106-x86_64-deb.tgz && cd mft-4.18.0-106-x86_64-deb
./install.sh

#Start MST service:
mst start
mst status
#copy the dev address with cr0 in it, like:
/dev/mst/mt4099_pci_cr0
#Use that address in the following commands

#Download latest FCBT firmware and unzip:
wget http://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip
unzip fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip

#Backup the cards current board definition file just in case
flint -d /dev/mst/mt4099_pci_cr0 dc orig_firmware.ini

#Flash the FCBT image to the card
flint -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin -allow_psid_change burn

Cold boot the server when done

Some useful config commands to config the card when it comes back up:

#start mst service again
mst start

#get detailed firmware info
mlxfwmanager --query

#get the device ID again:
mst status

#use it to see the cards current configuration:
mlxconfig -d /dev/mst/mt4099_pci_cr0 query

use it in config commands to configure the card
#for instance, to turn both ports from VPI/Auto to Ethernet only:
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2


#turn off bootrom crap
mlxconfig -d /dev/mst/mt4099_pci_cr0 set BOOT_OPTION_ROM_EN_P1=false
mlxconfig -d /dev/mst/mt4099_pci_cr0 set BOOT_OPTION_ROM_EN_P2=false
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LEGACY_BOOT_PROTOCOL_P1=0
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LEGACY_BOOT_PROTOCOL_P2=0


##optional: delete bootrom off the card, so it doesn't slow down boot by popping up crap
##this is safe to do and supported by mellanox
flint -d /dev/mst/mt4099_pci_cr0 --allow_rom_change drom

 

[Hindsight]

If you're just flashing a MCX354A-QCBT to a MCX354A-FCBT, you don't need to compile a firmware image - you can just flash the premade latest MCX354A-FCBT image from their site - this is nice because the latest firmware sources available for compiling are actually quite old.

When you use the firmware sources with the stock unedited FCBT ini file to compile a new image, this is all you're generating anyway. Except the pre-generated images on mellanox's site have been compiled with much newer firmware sources

Ohh great, that was a lot easier. Looks like it worked well. 2 more cards to flash. Thank you!

 

[Hindsight]

Is it possible to flash to ConnectX-3 Pro MCX354A-FCCT?

 

[arglebargle]

Is it possible to flash to ConnectX-3 Pro MCX354A-FCCT?

I'm interested in this as well -- Are the CX3 Pro cards different hardware or are the features just firmware locked?

Edit: I have two QCBT cards here now, I'm waiting on full height brackets and then I'll try flashing one to FCCT to test.

 

[arglebargle]

So no-go on the FCCT firmware, the firmware tools stop you from flashing the Pro firmware on a non Pro card.

Does anyone know if the IC on the pro is actually different or are they just differentiating SKUs to segment the market? If the hardware is identical I'd be happy to crack the firmware tool and try flashing Pro firmware on a non pro.

Also, I can't get my workstation to post with a CX-3 installed. This is the same machine I've had a CX-2 in for the last year, on the off chance this has happened to someone else is there any obvious reason a CX-3 would cause a Z97 board to hang on boot? I'm going to try again this afternoon after I swap cards around and update the firmware and disable PXE boot.