Flashing stock Mellanox firmware to OEM (EMC) ConnectX-3 IB/Ethernet dual-port QSFP adapter

nthu9280

Well-Known Member
Feb 3, 2016
1,593
448
83
San Antonio, TX
Thanks, I wonder why powershell worked
I got the error even in powershell when running in regular mode. Interesting thing is I was logged in as user that is a member of Admin group. I had to explicitly choose powershell admin mode and the query worked. I didn't try in cmd console as my deafault is set to powershell when I right click on Start.
 

Terry Wallace

PsyOps SysOp
Aug 13, 2018
151
89
28
52
Allen, TX
Probing pci hardware requires elevated powershell permissons unless you have specifically turned that off. If it's a standard windows install you'd have to use admin mode cmd/powershell to be able to directly probe hardware. Which is why it worked in elevated mode.
 
  • Like
Reactions: JustinClift

realtomatoes

Active Member
Oct 3, 2016
247
31
28
41
If you're just flashing a MCX354A-QCBT (649281-B21 on ebay) to a MCX354A-FCBT (full 40gbE enabled), you don't need to compile a firmware image - you can just flash the premade latest MCX354A-FCBT image from their site - this is nice because the latest firmware sources available for compiling are actually quite old.

When you use the firmware sources with the stock unedited FCBT ini file to compile a new image, this is all you're generating anyway. Except the pre-generated images on mellanox's site have been compiled with much newer firmware sources

from my notes:

Code:
##Download latest Mellanox Firmware tools and install them  + dependencies:

wget http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-deb.tgz
tar -xvf mft-4.10.0-104-x86_64-deb.tgz
cd mft-4.10.0-104-x86_64-deb/
apt-get install gcc make dkms unzip
apt install linux-headers-$(uname -r)
./install.sh

#Start MST service:
mst start
mst status
#copy the dev address with cr0 in it, like:
/dev/mst/mt4099_pci_cr0
#Use that address in the following commands

#Download latest FCBT firmware and unzip:
wget http://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip
unzip fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip

#Backup the cards current board definition file just in case
flint -d /dev/mst/mt4099_pci_cr0 dc orig_firmware.ini

#Flash the FCBT image to the card
flint -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin -allow_psid_change burn

Cold boot the server when done
Some useful config commands to config the card when it comes back up:

Code:
#start mst service again
mst start

#get detailed firmware info
mlxfwmanager --query

#get the device ID again:
mst status

#use it to see the cards current configuration:
mlxconfig -d /dev/mst/mt4099_pci_cr0 query

use it in config commands to configure the card
#for instance, to turn both ports from VPI/Auto to Ethernet only:
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2


#turn off bootrom crap
mlxconfig -d /dev/mst/mt4099_pci_cr0 set BOOT_OPTION_ROM_EN_P1=false
mlxconfig -d /dev/mst/mt4099_pci_cr0 set BOOT_OPTION_ROM_EN_P2=false
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LEGACY_BOOT_PROTOCOL_P1=0
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LEGACY_BOOT_PROTOCOL_P2=0


##optional: delete bootrom off the card, so it doesn't slow down boot by popping up crap
##this is safe to do and supported by mellanox
flint -d /dev/mst/mt4099_pci_cr0 --allow_rom_change drom
let me add my thanks to this great post.
flash worked without a hitch.
 

Luzer

New Member
Mar 1, 2015
9
1
3
35
Code:
##Download latest Mellanox Firmware tools and install them  + dependencies:

wget http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-deb.tgz
tar -xvf mft-4.10.0-104-x86_64-deb.tgz
cd mft-4.10.0-104-x86_64-deb/
apt-get install gcc make dkms unzip
apt install linux-headers-$(uname -r)
./install.sh

#Start MST service:
mst start
mst status
#copy the dev address with cr0 in it, like:
/dev/mst/mt4099_pci_cr0
#Use that address in the following commands

#Download latest FCBT firmware and unzip:
wget http://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip
unzip fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip

#Backup the cards current board definition file just in case
flint -d /dev/mst/mt4099_pci_cr0 dc orig_firmware.ini

#Flash the FCBT image to the card
flint -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin -allow_psid_change burn

Cold boot the server when done
Some useful config commands to config the card when it comes back up:

Code:
#start mst service again
mst start

#get detailed firmware info
mlxfwmanager --query

#get the device ID again:
mst status

#use it to see the cards current configuration:
mlxconfig -d /dev/mst/mt4099_pci_cr0 query

use it in config commands to configure the card
#for instance, to turn both ports from VPI/Auto to Ethernet only:
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2


#turn off bootrom crap
mlxconfig -d /dev/mst/mt4099_pci_cr0 set BOOT_OPTION_ROM_EN_P1=false
mlxconfig -d /dev/mst/mt4099_pci_cr0 set BOOT_OPTION_ROM_EN_P2=false
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LEGACY_BOOT_PROTOCOL_P1=0
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LEGACY_BOOT_PROTOCOL_P2=0


##optional: delete bootrom off the card, so it doesn't slow down boot by popping up crap
##this is safe to do and supported by mellanox
flint -d /dev/mst/mt4099_pci_cr0 --allow_rom_change drom
thanks, was able to reflash my two connectx-3.
 

Dave Corder

Active Member
Dec 21, 2015
126
53
28
38

DanielWood

Member
Sep 14, 2018
44
17
8
Does anyone know if these cards support the QSFP to SFP+ breakout cables? That would effectively give you 8x 10Gb connections from a single card, if it works at all.

I ask because it seems like a more economical way to get multiple 10G connections.
 

praetorian

New Member
Nov 16, 2016
1
0
1
34
Anyone know if there's a way to obtain and burn a UEFI ROM for these cards? I've scoured Mellanox's documentation and it's not clear if the ConnectX-3 IB/VPI cards support this (EN cards do, however). I'm mostly interested in this for getting UEFI PXE boot support.
 

RRB

New Member
Feb 5, 2019
7
7
3
Tried flashing HP 649281-B21 as per instructions however it seems the latest Mellanox WinMFT utility (4.11) gives error "specifying option flags after command is not allowed".

Can anyone help?
 

Terry Wallace

PsyOps SysOp
Aug 13, 2018
151
89
28
52
Allen, TX
Look at different examples for the command you’re trying to run. Or post it here. That error message means you have an option flag in the wrong place.
 

RRB

New Member
Feb 5, 2019
7
7
3
Here is the final command which is giving me difficulty:

Run:
0) mst status
1) mlxfwmanager
2) flint -d mt4099_pci_cr0 -i fw.bin burn -allow_psid_change

where "mt4099_pci_cr0" is the name output from commands 0 or 1

I referenced post #19 from this thread: https://forums.servethehome.com/ind...lanox-connectx-3-en-with-stock-firmware.8689/


Look at different examples for the command you’re trying to run. Or post it here. That error message means you have an option flag in the wrong place.
 
Last edited:

RRB

New Member
Feb 5, 2019
7
7
3
Did some digging and you're right, the original command is incorrect. You must move the burn command to the end.

For those who purchased an HP 649281-B21 and would like to flash it to stock MCX354A-FCBT firmware, proceed as follows:

For Windows Server 2016:

Step 1 - download and install:
Mellanox WinMFT_x64.exe (latest version from Mellanox Website under firmware tools)

Step 2 - download firmware for your card from Mellanox Website. Mine was an HP 649281-B21 which means i needed the MCX354A-FCBT firmware. Be sure which card is yours.
Rename downloaded firmware file to fw.bin
copy it to:
C:\Program Files\Mellanox\WinMFT>

Step 3 - open command prompt as administrator
cd to directory:
C:\Program Files\Mellanox\WinMFT>

Run:
0) mst status
1) mlxfwmanager
2) flint -d mt4099_pci_cr0 -i fw.bin -allow_psid_change burn

where "mt4099_pci_cr0" is the name output from commands 0 or 1

It should say "You are about to replace current PSID on flash - "HP_02802xxxxx" with a different PSID... etc.
Do you want to continue, type Y.

Thats it.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
656
234
43
Can i take a Brocade ICX6650 and utilize this as a 4 port 40GBe switch? In this way I truly only need 3 devices plugged into these ports for backups and job replications. I can then take the front part of these switches, drop in a bunch of 10GBE modules and call it day and have 10/40GBE?

I want to simplify my networking stack from 3 switches to 1 if possible with the least amount of equipment due to spacing. Right now it's a 1GB switch, 10GB switch, and 40GB switch. I have a feeling that the answer to my question is YES but it cannot hurt to double check.
I think you replied to the wrong tab, try here https://forums.servethehome.com/index.php?threads/brocade-icx-series-cheap-powerful-10gbe-40gbe-switching.21107/ :)

Though, yes, the 6650 has 4 usable 40Gb ports in addition to the two breakout ports.
 

Dreece

Active Member
Jan 22, 2019
506
154
43
Not to worry I found the info I wanted here:

https://forums.servethehome.com/index.php?threads/connectx-3-vs-connectx-3-pro.21381/

So basically the standard CX3 does not do RDMA over ethernet.

This means I must purchase the Pro variety in order to get similar functionality to my T5s... but of course Mellanox has up-to-date development support in latest Linux Azure kernel (the whole reason I'm making the transition).

A pity you gents hit a brick wall in the crossflashing project.

Now I must go and talk to the wallet. :D
 
Last edited:

Dreece

Active Member
Jan 22, 2019
506
154
43
I decided to just get a few current range Mellanox ConnectX-4s in the end. Hopefully serve my needs well for the foreseeable future.
 
Last edited:

Milos

New Member
Feb 22, 2018
23
1
3
Black hole with internet
I found a lot of info here but since lot of things are new to me I still don't know what to do with my cards. I am using Win 10 pro fresh install for flashing cards.
I got 3 cards all A3 an model cx354A.(form link where I order them SUN 7046442 DUAL 40GB/SEC QDR INFINIBAND CARD)
I have used WinMFT_x64_3_5_0_16 and MLNX_VPI_WinOF-5_50_52000_All_win2019_x64.
Idea is to flash normal firmware not SUNs.
When I tried burn command I got error , and I see there is not same PSID.Is it possible to do it this way or I am not doing something right.
I also put other card in other Win 10 machine and used QSFP/QSFP cable for direct connection.
As in pictures it shows connection but I can ping other address.

I am a little lost what to do know and how to flash cards.
 

Attachments

kiteboarder

Active Member
May 10, 2016
106
46
28
42
These are the notes I took down for myself when I did this a year or so back. YMMV. Please be careful using -allowpsidchange option.

Flashing firmware on HP 649281-B21 to stock Mellanox MCX354A-FCBT (ConnectX-3) guide:

For Windows Server 2016:

Step 1 - download and install:
Mellanox WinMFT_x64.exe (latest version from Mellanox Website under firmware tools)

Step 2 - download firmware for your card from Mellanox Website. Mine was an HP 649281-B21 which means i needed the MCX354A-FCBT firmware. Be sure which card is yours.
Rename downloaded firmware file to fw.bin
copy it to:
C:\Program Files\Mellanox\WinMFT>

Step 3 - open command prompt as administrator
cd to directory:
C:\Program Files\Mellanox\WinMFT>

Run:
0) mst status
1) mlxfwmanager
2) flint -d mt4099_pci_cr0 -i fw.bin -allow_psid_change burn

where "mt4099_pci_cr0" is the name output from commands 0 or 1

It should say "You are about to replace current PSID on flash - "HP_02802xxxxx" with a different PSID... etc.
Do you want to continue, type Y.