Flashing stock Mellanox firmware to OEM (EMC) ConnectX-3 IB/Ethernet dual-port QSFP adapter

[Hindsight]

Actually it seems the above is correct, A1 cards will not support it, but you have a much newer card, after they changed revision schema from A-x - it seems once they hit revision A5, the next revision up from that started at decimal 01 (no leading A) which is what you have - hence the date of 2017 - my A4 revision cards are as old as 2014

Ahh interesting. So then the Pro is most likely still different enough so it wouldn't run on my card.

 

[fohdeesha]

Ahh interesting. So then the Pro is most likely still different enough so it wouldn't run on my card.

yeah...that's what's worrying me. If it won't run on a card as new as yours (a 2k17 date code is the newest I've ever seen on a CX3 card) I'm worried it won't work on any. This card I have in a test box now is physically busted already as I previously mentioned so I think I'm going to go ahead and try it anyway with a couple different commands, will report back

 

[fohdeesha]

just a bit more double-checking - in the board configuration files for the cx3 FCBT vs CX3pro - the ASIC setup and clocking is identical between both - it really should work. trying to figure out why yours failed

 

[i386]

Maybe same asic (settings) but different cpus? Mellanox started using x86 on their switches when they released roce v2.

 

[arglebargle]

Maybe same asic (settings) but different cpus? Mellanox started using x86 on their switches when they released roce v2.

Ah, we're looking at the client cards here. As far as any of us can tell so far the hardware is almost identical on the CX-3 and CX-3 Pro adapters.

 

[fohdeesha]

bad news and good news: bad news, it doesn't work. I carefully flashed the CX3pro firmware, replaced the missing GUID and MAC strings since the raw flash erased these. Then did a dump of the onboard flash to verify it wrote the image to the correct location - it did, everything was exactly as it should be. However it just doesn't work - doesn't even get recognized, is totally bricked. So it seems the conclusive answer is you can NOT crossflash a CX3 to CX3pro - even though the ASIC is the same according to Mellanox stock brand images, I suspect they're actually 2 different variants.

Good news: Mellanox was smart enough to put a backup flash partition on the card, with just basic enough boot code to boot the card into "recovery mode" where you have access to the onboard flash and not much else. between thinking ahead like that, their open source firmware tools, the ability to crossflash pretty much everything they sell - you can tell Mellanox is ran by proper engineers - they have quickly become my favorite NIC vendor for many reasons.

Anyway, to recover from a brick @Hindsight you need to short out the 2 "flash present" pins on the card - this is how it knows to boot into recovery mode:

You can just short them out with some wire, or something, just make the two pads touch somehow. I ended up just soldering a little 2 pin header there, so I could use a standard mobo jumper to short them out:

Before putting it back in your server, take a pic of the rear sticker, or otherwise take note/write down the GUID and MAC values of the card (on the rear sticker). Then put the card back in your server and boot. then do the following to dump a working image back to flash:

#start the MST service
mst start

#get the device address, there will be just one pciconf device since it's in recovery
mst status

#use that address in the below command if it differs from the example
#get the GUID and MAC from the card, should be on rear sticker. stick them in the below command
#keep the leading "0x" before your values
#If you're lazy, use my values below
#If you use mine, don't sneak into my house and put that card onto my network or we'll have MAC collisions
#flash the MCX-354A-FCBT image back, might have to change the filename to cx3-fcbt.bin or change the command

flint -d /dev/mst/mt502_pciconf0 -i cx3-fcbt.bin -guid 0x24be05ffff8b97a0 -mac 0x24be058b97a1 -nofs burn

it will ask if a blank VSD is ok, choose yes (most are shipped with blank VSD's, perfectly normal - it's an optional OEM string). It'll then flash the image after asking you to verify a couple times.

when done, power the server off, remove the card, and remove your jumper. Put the card back in and boot normally - the card should be back and functional

 

[Hindsight]

I saw those pins last night and I was thinking of seeing what happened when I used them. I just went through your steps and it works great. My card is alive again with my ids. Thanks a lot for that. At least we know it can't be done.

Also seems like you don't need to use mst start anymore (least on windows)

>mst start
-E- There is no need to start/stop mst service anymore, it is done automatically by the tools

 

[arglebargle]

bad news and good news: bad news, it doesn't work. I carefully flashed the CX3pro firmware, replaced the missing GUID and MAC strings since the raw flash erased these. Then did a dump of the onboard flash to verify it wrote the image to the correct location - it did, everything was exactly as it should be. However it just doesn't work - doesn't even get recognized, is totally bricked. So it seems the conclusive answer is you can NOT crossflash a CX3 to CX3pro - even though the ASIC is the same according to Mellanox stock brand images, I suspect they're actually 2 different variants.

Thanks a lot for testing this, I knew it was a long shot but with how Mellanox differentiates SKUs I figured it was worth at least attempting.

 

[tommybackeast]

If you're just flashing a MCX354A-QCBT to a MCX354A-FCBT, you don't need to compile a firmware image - you can just flash the premade latest MCX354A-FCBT image from their site - this is nice because the latest firmware sources available for compiling are actually quite old.

When you use the firmware sources with the stock unedited FCBT ini file to compile a new image, this is all you're generating anyway. Except the pre-generated images on mellanox's site have been compiled with much newer firmware sources

from my notes:

##Download latest Mellanox Firmware tools and install them  + dependencies:

wget http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-deb.tgz
tar -xvf mft-4.10.0-104-x86_64-deb.tgz
cd mft-4.10.0-104-x86_64-deb/
apt-get install gcc make dkms unzip
apt install linux-headers-$(uname -r)
./install.sh

#Start MST service:
mst start
mst status
#copy the dev address with cr0 in it, like:
/dev/mst/mt4099_pci_cr0
#Use that address in the following commands

#Download latest FCBT firmware and unzip:
wget http://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip
unzip fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip

#Backup the cards current board definition file just in case
flint -d /dev/mst/mt4099_pci_cr0 dc orig_firmware.ini

#Flash the FCBT image to the card
flint -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin -allow_psid_change burn

Cold boot the server when done

Some useful config commands to config the card when it comes back up:

#start mst service again
mst start

#get detailed firmware info
mlxfwmanager --query

#get the device ID again:
mst status

#use it to see the cards current configuration:
mlxconfig -d /dev/mst/mt4099_pci_cr0 query

use it in config commands to configure the card
#for instance, to turn both ports from VPI/Auto to Ethernet only:
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2


#turn off bootrom crap
mlxconfig -d /dev/mst/mt4099_pci_cr0 set BOOT_OPTION_ROM_EN_P1=false
mlxconfig -d /dev/mst/mt4099_pci_cr0 set BOOT_OPTION_ROM_EN_P2=false
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LEGACY_BOOT_PROTOCOL_P1=0
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LEGACY_BOOT_PROTOCOL_P2=0

Can one also achieve this via MLNX's Windows program? or must it be done under linux as you've shown?

 

[Hindsight]

Can one also achieve this via MLNX's Windows program? or must it be done under linux as you've shown?

I did it using windows, no issues.

 

[fohdeesha]

yeah, you just download the mellanox MFT installer for windows, and run the commands from command prompt, commands don't change (just skip the linux specific install stuff)

 

[tommybackeast]

I did it using windows, no issues.

Thank you.

Either I was having a stupid moment, or the Mellanox windows MFT installer program was difficult to find.

Mellanox - MFT Downloader

I am used to Windows : insert the PCIe device in a windows PC; flash the firmware, reboot; and the PCIe device stays in that same PC.

In this case, I will be using this card in a Dell Tower Server, and most likely using ESXi hypervisor.

But i'm new to ESXi. so can I use Mellanox MFT windows installer; flash the firmware; and then insert the PCIe card into the Dell Server -or- must I flash the firmware under the Dell Server ESXi method?

 

[Hindsight]

No don't try and do it in esxi, you'd have to try and pass it through to a windows VM. It's easier to put it in your desktop and flash then them put it in your server. I had it setup on my pc, and did 3 cards in a row which went to my hp and dell servers.

 

[tommybackeast]

No don't try and do it in esxi, you'd have to try and pass it through to a windows VM. It's easier to put it in your desktop and flash then them put it in your server. I had it setup on my pc, and did 3 cards in a row which went to my hp and dell servers.

Wonderful; thank you so much for explaining that process.

 

[PGlover]

yeah, you just download the mellanox MFT installer for windows, and run the commands from command prompt, commands don't change (just skip the linux specific install stuff)

I will be getting my cards tomorrow and plan to do the flashing in Windows. When you run the "mst status" command in Windows will the device address show up as "mt4099_pci_cr0" or "/dev/mst/mt4099_pci_cr0"? I understand that my device address may be different, but wanted to make sure what syntax of the device address that I need to use in sequence commands in Windows.

 

[mb300sd]

I will be getting my cards tomorrow and plan to do the flashing in Windows. When you run the "mst status" command in Windows will the device address show up as "mt4099_pci_cr0" or "/dev/mst/mt4099_pci_cr0"? I understand that my device address may be different, but wanted to make sure what syntax of the device address that I need to use in sequence commands in Windows.

It'll be mt4099_pci_cr0 in Windows. Run "mst status" to list devices.

 

[PGlover]

The flashing went well in Windows; however, the following command did not work in Windows.

mlxconfig -d mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

Here is the error message I keep getting. I know the device address is correct because I was able to flash the unit using that device address.

 

[Hindsight]

You can also use the snipping tool so you don't have to take a picture of your screen.

Did you reboot after you flashed? Also I don't really see a reason to change it. Leave it in VPI mode and you can use both infiniband and/or ethernet. I'm only using ethernet but left in VPI with no issues.

 

[fohdeesha]

try using the pciconf address, and yeah, if you just flashed, you need to reboot to re-initialize the card with the new firmware first

 

[PGlover]

try using the pciconf address, and yeah, if you just flashed, you need to reboot to re-initialize the card with the new firmware first

I rebooted and I tried the pciconf address as well...