More custom Mellanox firmware tips

J Hart

Active Member
Apr 23, 2015
144
100
43
40
I saw this article from a couple of years ago and wanted to share my experience building a custom firmware to get a newer revision.

The original article dealt with the tools for Windows, but I operate mostly in Linux so I'll describe those tools. First, you need to acquire all of the tools and drivers. Every kernel I have used in recent memory included the mlx4 modules by default, but many don't load mlx4_ib or ib_umad or ib_uverbs. Load these(modprobe mlx4_ib) before you get started. Next you will probably want to have infiniband_diags as this package includes ibstat. You'll also want mstflint

Once you have all the tools, find the PCI locations of the card
dmesg | grep mlx
Code:
[   18.254192] mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014)
[   18.323748] mlx4_core: Initializing 0000:13:00.0
[   20.798217] mlx4_core 0000:13:00.0: PCIe link speed is 5.0GT/s, device supports 5.0GT/s
[   20.798218] mlx4_core 0000:13:00.0: PCIe link width is x8, device supports x8
[   20.885096] mlx4_core: Initializing 0000:19:00.0
[   23.148705] mlx4_core 0000:19:00.0: PCIe link speed is 5.0GT/s, device supports 5.0GT/s
[   23.148707] mlx4_core 0000:19:00.0: PCIe link width is x8, device supports x8
In my case I have 2 cards a 10GbE and a QDR Infiniband card. I'll update the Infiniband card

sudo mstflint -d 19:00.0 q
Code:
Image type:      FS2
FW Version:      2.9.1000
FW Release Date: 5.6.2011
Device ID:       26428
Description:     Node             Port1            Port2            Sys image
GUIDs:           00212800013f118a 00212800013f118b 00212800013f118c 00212800013f118d
MACs:                                 000000000000     000000000001
VSD:             
PSID:            SUN0150000009
So this card is a Sun rebranded Mellanox card.

We need to get the ini file to produce a new firmware:
sudo mstflint -d 19:00.0 dc > SUN0150000009.ini

The type of card is key to finding the correct firmware to download. You can get a pretty big hint from the PCI information:
lspci | grep Mellanox
Code:
13:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
19:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev a0)
You'll have to figure out what sort of card you have to download the appropriate firmware from Mellanox. In my case it just says ConnectX and the chip type is MT26428 revision a0. This along with some googling of the part number led me to the Mellanox part number of MHQH29-XSR. Go to the custom firmware page and find the card.

Download the MFT package from mellanox and install it so that we have the mlxburn tool. Then run this command to generate a new firmware image.
sudo mlxburn -wrimage sun.img -fw fw-25408-rel.mlx -conf SUN0150000009.ini

And now you can write this to the card, but first we should back up the original so we can revert if needed

sudo mstflint -d 19:00.0 ri sun-old.img

Finally write it to the card:
sudo mstflint -d 19:00.0 -i sun.img b

You can check that that burn was successful by querying the card
sudo mstflint -d 19:00.0 q

Reboot to load the new firmware.

Anyway I found that the original article didn't work for me. flint refused to read the info off of the card saying it was an unsupported device.
 

mpogr

Active Member
Jul 14, 2016
113
81
28
50
Hi all,

A question related to this subject: I need to flash a device with a firmware which is packed inside an MFA archive. These archives can be flashed using mlxfwmanager, but not flint/mstflint. The problem is the device has a different PSID, but mlxfwmanager doesn't have the overriding options like flint.
What would be the best way to do this? I know MFA files are just custom Mellanox archives, which can be reconstructed, as mlx_mfa_gen tool is written in Perl. But is there a simpler way?
Another problem is I don't know the right PSID for the BIN I need to flash, and the archive has many of them. I'm pretty sure it's kept somewhere inside the the BIN. What's the easiest way to identify the right BIN having just the device name/part number?

Thanks!
 

J Hart

Active Member
Apr 23, 2015
144
100
43
40
I think the easiest way would be to use lspci -nn which will give you both the Chipset, revision and the PCI ID number. Those chipset numbers and PCI numbers don't change even for OEM PSIDs. Now how to go from those numbers to the particular model of card... Yeah that is the tricky bit. There isn't a super easy way. I'd suggest just having a look at the card especially the ports and then looking at the mellanox products for the same family of chips.

In my case I get this from lspci -nn
Code:
19:00.0 InfiniBand [0c06]: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] [15b3:673c] (rev a0)
It says it is the first gen ConnectX. Looking at the card it has the short bracket and 2 QSFP ports. So I go to the Mellanox ConnectX firmware page and 3 have dual QSFP ports. Only one has a short bracket, so that gives us MHQH29-XSC and the firmware package. Even if the PSID doesn't match you can make a firmware image that will have your PSID by using the instructions above. You can tell flint/mstflint to ignore the PSID when flashing, but this might be a bad idea. In any case backup the firmware and bootrom just in case so you can revert if things go really wrong.

Hi all,

A question related to this subject: I need to flash a device with a firmware which is packed inside an MFA archive. These archives can be flashed using mlxfwmanager, but not flint/mstflint. The problem is the device has a different PSID, but mlxfwmanager doesn't have the overriding options like flint.
What would be the best way to do this? I know MFA files are just custom Mellanox archives, which can be reconstructed, as mlx_mfa_gen tool is written in Perl. But is there a simpler way?
Another problem is I don't know the right PSID for the BIN I need to flash, and the archive has many of them. I'm pretty sure it's kept somewhere inside the the BIN. What's the easiest way to identify the right BIN having just the device name/part number?

Thanks!
 

mpogr

Active Member
Jul 14, 2016
113
81
28
50
The situation is a bit complicated here. The device I'm trying to flash is an SX6012 switch. It would have all been good if the switch had been a genuine Mellanox one. However, I bought it on eBay (almost new for a really good price of ~450 USD including postage from Ireland to Australia) and it turned out to be the EMC variant of the switch. It has the same hardware, but the software is completely different, i.e. it doesn't run MLNX-OS.

At first, I managed to burn the SX6005 firmware onto it, effectively making it a non-managed switch, and it was working fine. However, I was missing the built-in Subnet Manager as well as the ability to use some of the ports in Ethernet mode, so I've been working on hacking MLNX-OS to make it work on that switch. Now, I almost managed to accomplish it, but there is still a problem with the SM being unable to fully communicate through the ports, and I suspect the firmware could be the reason.

The usual means (via the Web UI and the CLI) provided by Mellanox to flash firmware on those switches effectively invoke mlxfwmanager with an MFA file provided within the MLNX-OS package. However, as I mentioned previously, it fails, because the PSID on the switch doesn't match the one inside the MFA.

If I had been able to:
a) break the MFA down into the individual BINs and
b) identify the right BIN

I would be able to use flint to burn it. So I thought some of the guys here might have an idea on how to do that.

Just FYI, Mellanox doesn't provide SX6012 (or any other current managed switches) firmware in any other form, and it doesn't even have information about these switches PSIDs on their Web site.
 

J Hart

Active Member
Apr 23, 2015
144
100
43
40
Here are some for the SX6005/SX6012 I have no idea if they will work for your switch or not. That MFA file could possibly be cracked open using binwalk. It will probably take some fiddling, but I have used that utility previously with other switch firmware to get at the internals.
 

mpogr

Active Member
Jul 14, 2016
113
81
28
50
Yeah mate, I've seen that page... It has firmwares/PSIDs only for the unmanaged versions (XXX5) of those switches. I'm now 90% convinced that having the right firmware on the chip would let me make the last step through the MLNX-OS hacking exercise. I've already gone pretty long way (modifying some MLNX-OS binaries using the PPC disassembler) to get here, I guess I'll have to take the plunge and crack those MFAs open as well...
 
  • Like
Reactions: hardman

mpogr

Active Member
Jul 14, 2016
113
81
28
50
OK, mission accomplished. The MFA file I had turned out to be really easy to crack. It has a header and 3 consequently appended .xz archives. The first one includes all the PSIDs and switch descriptions, which made identifying the right .bin really easy. The third one is just an archive of all the .bins concatenated into a single file. After unxz-ing (with the --single-stream flag), identifying the .bin boundaries is very easy, because they start with a distinct header. So I ended up with the right .bin file (verified by running "flint -i <file> q"), which I successfully flashed into the switch using the --allow_psid_change flag to make subsequent updates by mlxfwmanager automatic and painless.
 
  • Like
Reactions: hardman and J Hart
Apr 21, 2016
47
17
8
40
@mpogr - let me get this straight - you can download the mfa via online update with mlxfwmanager/mlxup/mlxburn and then is just a matter of pushing the correct binary image to the switch ?
can you push an image that has management to a unmanaged switch ?
 

mpogr

Active Member
Jul 14, 2016
113
81
28
50
@mpogr - let me get this straight - you can download the mfa via online update with mlxfwmanager/mlxup/mlxburn and then is just a matter of pushing the correct binary image to the switch ?
can you push an image that has management to a unmanaged switch ?
You're confusing between firmware (which runs on the Infiniband board of the switch) and software running on the managed switch processor (the unmanaged switches don't have it).
In this particular case, yes, you can push firmware of the unamanged switch model to the analogous managed switch, essentially making it unmanaged. However, doing the opposite (which you're asking about) won't help at all, as the managed switch firmware relies on software running on the managed switch CPU, which the unmanaged switch won't have. You're very likely to brick it this way.
 

whitey

Moderator
Jun 30, 2014
2,770
866
113
38
For grins I used this process after grabbing the intermediate update. I should probably list more background/details but PM me if you get stuck.

Used OFED 3.4-1
lspci | grep -i mel
mstflint -d 03:00.0 q (query PCI dev ID, may be different for your system)
Download intermediate FW here: http://www.mellanox.com/page/custom_firmware_table

THESE WERE THE KEY CMDS I had to dig up/scour the web for
mlxburn -fw fw-ConnectX2-rel.mlx -conf MHQH19B-XTR_A1-A3.ini -wrimage fw-ConnectX2-rel-2_9_1200-MHQH19B-XTR_A1-A3.bin
THEN
mstflint -d 03:00.0 -i fw-ConnectX2-rel-2_9_1200-MHQH19B-XTR_A1-A3.bin b

EDIT: Sorry @J Hart I didn't read the orig thread throughly, you covered this process quite excellent, leaving here for prosperity sake though! This is one of those 'If I could give ya 5 likes I would'...Good stuff!
 

TeleFragger

Member
Oct 26, 2016
147
19
18
47
all the links for custom firmware are gone...
I have an old card..
and while it is an HP, I found HP's latest firmware to be 2.8.

I got the cards working in win10 doing 10gb eth file copy and reading saw something that 2.9.1200 enables.
latest I could find for the "card" non hp was 2.9.1000 - gave it a whirl with force flash and it took and card still seem to be working.

still want to learn how to create a 2.9.1200 firmware if anyone can point me...
oh and in windows so the commands above don't make senst to me.. LOL

so I flashed fw-25408-2_8_0000-483514-B21.bin
with fw-25408-2_9_1000-MHGH29-XTC_A1.bin

so HP p/n 483514-B21
Mellanox card MHGH29-XTC
 

Cifu

New Member
Mar 25, 2019
1
0
1
Same problem for me.
I just got a ConnectX-3 Pro (MCX-312C) - PSID: EMC0000000002

And I cannot make the card to work.
The lshw -class network say:

*-network UNCLAIMED

The Mellanox MFT say:

"The firmware for this device is not distributed inside Mellanox driver: 01:00.0 (PSID: EMC0000000002)
To obtain firmware for this device, please contact your HW vendor."

The custom firmware section are gone from the Mellanox homepage, now the firmware flashing become a bit harder than earlier...
 

BeTeP

Well-Known Member
Mar 23, 2019
521
324
63
A question related to this subject: I need to flash a device with a firmware which is packed inside an MFA archive. These archives can be flashed using mlxfwmanager, but not flint/mstflint. The problem is the device has a different PSID, but mlxfwmanager doesn't have the overriding options like flint.
What would be the best way to do this? I know MFA files are just custom Mellanox archives, which can be reconstructed, as mlx_mfa_gen tool is written in Perl. But is there a simpler way?
Another problem is I don't know the right PSID for the BIN I need to flash, and the archive has many of them. I'm pretty sure it's kept somewhere inside the the BIN. What's the easiest way to identify the right BIN having just the device name/part number?
@mpogr has solved his problem long time ago but I found this thread just now. Here's my take on the .mfa unpacking:
Code:
#!/bin/bash

# requires
# apt install -y gawk xz-utils xxd mstflint

LANG=C
VER="9_4_5090"
MFA="fw-SX-rel-${VER}-FIT.mfa"
PSID="MT_1270110020"

mkdir -p ".$MFA"

# get offsets of 3 data streams inside of .mfa
OFFSETS=($(grep -obaP "\xFD7zXZ" $MFA | awk -F: 'NR<4{print$1}'))

# extract the streams
for I in 0 1 2; do dd if=$MFA bs=${OFFSETS[I]} skip=1 2>/dev/null | xzcat --single-stream --ignore-check > .$MFA/file$I; done

# list all included PSIDs. optional
#cat .$MFA/file0 | tr '[\0-\7\200-\377]' '[ *]' | sed -e 's/.\{228\}/&\n/g' -e 's/ *PN /\t/g' -e 's/ DESCRIPTION /\t\t\t/g' | cat -n

# get index of the desired PSID
IDX=$(sed 's/\(.\{13\}\)\(.\{215\}\)/\1\n/g' .$MFA/file0 | awk '"'$PSID'"==$1{print NR;exit}')

# get offset and length of the PSID firmware. this requires GNU Awk
declare $(xxd -g4 -c24 .$MFA/file1 | awk 'NR=='$IDX'{printf"SKIP=%d\nCOUNT=%d\n",strtonum("0x"$2),strtonum("0x"$3)}')

# extract .bin firmware for the PSID
dd if=.$MFA/file2 of=$PSID-$VER.bin bs=1 count=$COUNT skip=$SKIP 2>/dev/null

# check firmware
mstflint -i $PSID-$VER.bin q

# delete temporary files
rm -rf ".$MFA"

exit 0
 
Last edited:

Derwood

Member
May 22, 2019
48
0
6
Just wanted to add to this as I've been searching after reading the same and a number of other articles.

I myself have the SUN ConnectX-2 dual 40GBPS QDR cards and have upgraded the firmware from 2.9.x up to 2.11.x if anyone would like to seek higher as I'm sure listed within one of the documents at least 2.10.x was required to even operate RDMA via IB within windows server.

I'll come back in a while and upload the packages if I'm allowed.
 

Falloutboy

Member
Oct 23, 2011
207
19
18
Just wanted to add to this as I've been searching after reading the same and a number of other articles.

I myself have the SUN ConnectX-2 dual 40GBPS QDR cards and have upgraded the firmware from 2.9.x up to 2.11.x if anyone would like to seek higher as I'm sure listed within one of the documents at least 2.10.x was required to even operate RDMA via IB within windows server.

I'll come back in a while and upload the packages if I'm allowed.
I have to ask don't sun cards following their own numbering practices? if relating this firmware back to actual Mellanox Connect X series product, what products does it relate to and is the sun firmware later or earlier by comparison @Derwood
 
  • Like
Reactions: Derwood

ananyagupta

New Member
Oct 19, 2020
1
0
1
Hi, Thanks for sharing informative information with us about firmware activate RDMA, which I don't have an idea before.
 

Derwood

Member
May 22, 2019
48
0
6
I have to ask don't sun cards following their own numbering practices? if relating this firmware back to actual Mellanox Connect X series product, what products does it relate to and is the sun firmware later or earlier by comparison @Derwood
Yea these SUN cards are or at least have these numbering practices surrounding them MHQH29B-XSR & 375-3696-01 I reckon. And the SUN firmware is earlier, I've risen x.2.x it's not much but given me a linear FW across the cards and RDMA, IPoIB additions.

I'm not going to stand here at the rack and tell you everything is streamlined, it's not in production usage and still working it way to being a finished network in total. But it's going along okay at present.