Flashing stock Mellanox firmware to OEM (EMC) ConnectX-3 IB/Ethernet dual-port QSFP adapter

[nthu9280]

Thanks, I wonder why powershell worked

I got the error even in powershell when running in regular mode. Interesting thing is I was logged in as user that is a member of Admin group. I had to explicitly choose powershell admin mode and the query worked. I didn't try in cmd console as my deafault is set to powershell when I right click on Start.

 

[Terry Wallace]

Probing pci hardware requires elevated powershell permissons unless you have specifically turned that off. If it's a standard windows install you'd have to use admin mode cmd/powershell to be able to directly probe hardware. Which is why it worked in elevated mode.

 

[realtomatoes]

If you're just flashing a MCX354A-QCBT (649281-B21 on ebay) to a MCX354A-FCBT (full 40gbE enabled), you don't need to compile a firmware image - you can just flash the premade latest MCX354A-FCBT image from their site - this is nice because the latest firmware sources available for compiling are actually quite old.

When you use the firmware sources with the stock unedited FCBT ini file to compile a new image, this is all you're generating anyway. Except the pre-generated images on mellanox's site have been compiled with much newer firmware sources

from my notes:

##Download latest Mellanox Firmware tools and install them  + dependencies:

wget http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-deb.tgz
tar -xvf mft-4.10.0-104-x86_64-deb.tgz
cd mft-4.10.0-104-x86_64-deb/
apt-get install gcc make dkms unzip
apt install linux-headers-$(uname -r)
./install.sh

#Start MST service:
mst start
mst status
#copy the dev address with cr0 in it, like:
/dev/mst/mt4099_pci_cr0
#Use that address in the following commands

#Download latest FCBT firmware and unzip:
wget http://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip
unzip fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip

#Backup the cards current board definition file just in case
flint -d /dev/mst/mt4099_pci_cr0 dc orig_firmware.ini

#Flash the FCBT image to the card
flint -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin -allow_psid_change burn

Cold boot the server when done

Some useful config commands to config the card when it comes back up:

#start mst service again
mst start

#get detailed firmware info
mlxfwmanager --query

#get the device ID again:
mst status

#use it to see the cards current configuration:
mlxconfig -d /dev/mst/mt4099_pci_cr0 query

use it in config commands to configure the card
#for instance, to turn both ports from VPI/Auto to Ethernet only:
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2


#turn off bootrom crap
mlxconfig -d /dev/mst/mt4099_pci_cr0 set BOOT_OPTION_ROM_EN_P1=false
mlxconfig -d /dev/mst/mt4099_pci_cr0 set BOOT_OPTION_ROM_EN_P2=false
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LEGACY_BOOT_PROTOCOL_P1=0
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LEGACY_BOOT_PROTOCOL_P2=0


##optional: delete bootrom off the card, so it doesn't slow down boot by popping up crap
##this is safe to do and supported by mellanox
flint -d /dev/mst/mt4099_pci_cr0 --allow_rom_change drom

let me add my thanks to this great post.
flash worked without a hitch.

 

[Luzer]

##Download latest Mellanox Firmware tools and install them  + dependencies:

wget http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-deb.tgz
tar -xvf mft-4.10.0-104-x86_64-deb.tgz
cd mft-4.10.0-104-x86_64-deb/
apt-get install gcc make dkms unzip
apt install linux-headers-$(uname -r)
./install.sh

#Start MST service:
mst start
mst status
#copy the dev address with cr0 in it, like:
/dev/mst/mt4099_pci_cr0
#Use that address in the following commands

#Download latest FCBT firmware and unzip:
wget http://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip
unzip fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip

#Backup the cards current board definition file just in case
flint -d /dev/mst/mt4099_pci_cr0 dc orig_firmware.ini

#Flash the FCBT image to the card
flint -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin -allow_psid_change burn

Cold boot the server when done

Some useful config commands to config the card when it comes back up:

#start mst service again
mst start

#get detailed firmware info
mlxfwmanager --query

#get the device ID again:
mst status

#use it to see the cards current configuration:
mlxconfig -d /dev/mst/mt4099_pci_cr0 query

use it in config commands to configure the card
#for instance, to turn both ports from VPI/Auto to Ethernet only:
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2


#turn off bootrom crap
mlxconfig -d /dev/mst/mt4099_pci_cr0 set BOOT_OPTION_ROM_EN_P1=false
mlxconfig -d /dev/mst/mt4099_pci_cr0 set BOOT_OPTION_ROM_EN_P2=false
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LEGACY_BOOT_PROTOCOL_P1=0
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LEGACY_BOOT_PROTOCOL_P2=0


##optional: delete bootrom off the card, so it doesn't slow down boot by popping up crap
##this is safe to do and supported by mellanox
flint -d /dev/mst/mt4099_pci_cr0 --allow_rom_change drom

thanks, was able to reflash my two connectx-3.

 

[Dave Corder]

Another pair of HP cards flashed to the FCBT firmware here. Now if only my full-height brackets would finally show up...

 

[Jannis Jacobsen]

Another pair of HP cards flashed to the FCBT firmware here. Now if only my full-height brackets would finally show up...

or you can just 3dprint them
Full Height Bracket for QSFP PCIe Network Adapter (HP/Mellanox 649281-B21 Infiniband) by samuelpinches

-j

 

[Dave Corder]

or you can just 3dprint them
Full Height Bracket for QSFP PCIe Network Adapter (HP/Mellanox 649281-B21 Infiniband) by samuelpinches

-j

Well, that would have been useful earlier today. Turns out I accidentally ordered the wrong brackets (for a dual SFP rather than dual QSFP card), but a few minutes with my Dremel this afternoon resolved the issue well enough

 

[DanielWood]

Does anyone know if these cards support the QSFP to SFP+ breakout cables? That would effectively give you 8x 10Gb connections from a single card, if it works at all.

I ask because it seems like a more economical way to get multiple 10G connections.

 

[fohdeesha]

Nope, only the switches support breakout

 

[praetorian]

Anyone know if there's a way to obtain and burn a UEFI ROM for these cards? I've scoured Mellanox's documentation and it's not clear if the ConnectX-3 IB/VPI cards support this (EN cards do, however). I'm mostly interested in this for getting UEFI PXE boot support.

 

[RRB]

Tried flashing HP 649281-B21 as per instructions however it seems the latest Mellanox WinMFT utility (4.11) gives error "specifying option flags after command is not allowed".

Can anyone help?

 

[Terry Wallace]

Look at different examples for the command you’re trying to run. Or post it here. That error message means you have an option flag in the wrong place.

 

[RRB]

Here is the final command which is giving me difficulty:

Run:
0) mst status
1) mlxfwmanager
2) flint -d mt4099_pci_cr0 -i fw.bin burn -allow_psid_change

where "mt4099_pci_cr0" is the name output from commands 0 or 1

I referenced post #19 from this thread: https://forums.servethehome.com/ind...lanox-connectx-3-en-with-stock-firmware.8689/

Look at different examples for the command you’re trying to run. Or post it here. That error message means you have an option flag in the wrong place.

 

[RRB]

Did some digging and you're right, the original command is incorrect. You must move the burn command to the end.

For those who purchased an HP 649281-B21 and would like to flash it to stock MCX354A-FCBT firmware, proceed as follows:

For Windows Server 2016:

Step 1 - download and install:
Mellanox WinMFT_x64.exe (latest version from Mellanox Website under firmware tools)

Step 2 - download firmware for your card from Mellanox Website. Mine was an HP 649281-B21 which means i needed the MCX354A-FCBT firmware. Be sure which card is yours.
Rename downloaded firmware file to fw.bin
copy it to:
C:\Program Files\Mellanox\WinMFT>

Step 3 - open command prompt as administrator
cd to directory:
C:\Program Files\Mellanox\WinMFT>

Run:
0) mst status
1) mlxfwmanager
2) flint -d mt4099_pci_cr0 -i fw.bin -allow_psid_change burn

where "mt4099_pci_cr0" is the name output from commands 0 or 1

It should say "You are about to replace current PSID on flash - "HP_02802xxxxx" with a different PSID... etc.
Do you want to continue, type Y.

Thats it.

 

[arglebargle]

Can i take a Brocade ICX6650 and utilize this as a 4 port 40GBe switch? In this way I truly only need 3 devices plugged into these ports for backups and job replications. I can then take the front part of these switches, drop in a bunch of 10GBE modules and call it day and have 10/40GBE?

I want to simplify my networking stack from 3 switches to 1 if possible with the least amount of equipment due to spacing. Right now it's a 1GB switch, 10GB switch, and 40GB switch. I have a feeling that the answer to my question is YES but it cannot hurt to double check.

I think you replied to the wrong tab, try here https://forums.servethehome.com/index.php?threads/brocade-icx-series-cheap-powerful-10gbe-40gbe-switching.21107/

Though, yes, the 6650 has 4 usable 40Gb ports in addition to the two breakout ports.

 

[Dreece]

Not to worry I found the info I wanted here:

https://forums.servethehome.com/index.php?threads/connectx-3-vs-connectx-3-pro.21381/

So basically the standard CX3 does not do RDMA over ethernet.

This means I must purchase the Pro variety in order to get similar functionality to my T5s... but of course Mellanox has up-to-date development support in latest Linux Azure kernel (the whole reason I'm making the transition).

A pity you gents hit a brick wall in the crossflashing project.

Now I must go and talk to the wallet.

 

[i386]

So basically the standard CX3 does not do RDMA over ethernet.

Rdma works fine between Windows Servers with the cx3s

 

[Dreece]

I decided to just get a few current range Mellanox ConnectX-4s in the end. Hopefully serve my needs well for the foreseeable future.

 

[Milos]

I found a lot of info here but since lot of things are new to me I still don't know what to do with my cards. I am using Win 10 pro fresh install for flashing cards.
I got 3 cards all A3 an model cx354A.(form link where I order them SUN 7046442 DUAL 40GB/SEC QDR INFINIBAND CARD)
I have used WinMFT_x64_3_5_0_16 and MLNX_VPI_WinOF-5_50_52000_All_win2019_x64.
Idea is to flash normal firmware not SUNs.
When I tried burn command I got error , and I see there is not same PSID.Is it possible to do it this way or I am not doing something right.
I also put other card in other Win 10 machine and used QSFP/QSFP cable for direct connection.
As in pictures it shows connection but I can ping other address.

I am a little lost what to do know and how to flash cards.

 

[kiteboarder]

These are the notes I took down for myself when I did this a year or so back. YMMV. Please be careful using -allowpsidchange option.

Flashing firmware on HP 649281-B21 to stock Mellanox MCX354A-FCBT (ConnectX-3) guide:

For Windows Server 2016:

Step 1 - download and install:
Mellanox WinMFT_x64.exe (latest version from Mellanox Website under firmware tools)

Step 2 - download firmware for your card from Mellanox Website. Mine was an HP 649281-B21 which means i needed the MCX354A-FCBT firmware. Be sure which card is yours.
Rename downloaded firmware file to fw.bin
copy it to:
C:\Program Files\Mellanox\WinMFT>

Step 3 - open command prompt as administrator
cd to directory:
C:\Program Files\Mellanox\WinMFT>

Run:
0) mst status
1) mlxfwmanager
2) flint -d mt4099_pci_cr0 -i fw.bin -allow_psid_change burn

where "mt4099_pci_cr0" is the name output from commands 0 or 1

It should say "You are about to replace current PSID on flash - "HP_02802xxxxx" with a different PSID... etc.
Do you want to continue, type Y.