Flash OEM (IBM) Mellanox ConnectX-3 EN with stock firmware?

Stereodude

Active Member
Feb 21, 2016
412
65
28
USA
So I picked up an IBM flavored Mellanox ConnectX-3 EN (MCX312A-XCBT / Dual 10GbE SFP+) from eBay. It's FRU: 00D9692. The latest WinOF driver installed without issue (Windows 7 Pro).

The card has a PSID of IBM1080111023 so the standard MT_1080110023 firmware won't load on it. Here's what I see:

------------------------------------------------------------------------
C:\Program Files\Mellanox\WinMFT>flint -d mt4099_pci_cr0 -i fw-ConnectX3-rel-2_34_5000-MCX312A-XCB_A1-FlexBoot-3.4.521.bin burn

Current FW version on flash: 2.30.3200
New FW version: 2.34.5000

-E- PSID mismatch. The PSID on flash (IBM1080111023) differs from the PSID in the given image (MT_1080110023).
------------------------------------------------------------------------

Is there a way to flash the generic firmware? I saw this guide but I'm not sure if I can apply it to my case or not. The firmware I have is a .bin, not a .mlx, and I'm not sure about where to get an equivalent .ini file.

Has anyone tried this or know how it can be done?
 
Last edited:

Stereodude

Active Member
Feb 21, 2016
412
65
28
USA
I guess I should have looked a little harder. I found this: http://www.mellanox.com/page/custom_firmware_table

However, the ConnectX-3 EN download package is for rel-2.36.5000, and the firmware page shows rel-2.34.5000 for the MT_1080110023. I'm not sure whether the IBM1080111023 is a MT_1080110023 or a MT_1080120023...

The sticker on the back of the card says Rev. A5 and was made in 2014-03-27 which would seem to suggest it might be equivalent to a MT_1080120023, but I'm not sure.

Since I don't have my twinax cable yet or anything to plug this into, I guess I will sit tight for a while. Hopefully someone can shed some light on this.
 
Last edited:

Stereodude

Active Member
Feb 21, 2016
412
65
28
USA
So someone on another forum pointed out the mlxup utility for Mellanox cards. That brought the card up from 2.30.3200 to 2.33.5000. Here's what I saw (MAC addresses removed by me):

Code:
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:  ConnectX3
  Part Number:  00D9691_Ax
  Description:  Mellanox ConnectX-3 EN Dual-port SFP+ 10GbE Adapter
  PSID:  IBM1080111023
  PCI Device Name:  mt4099_pci_cr0
  Port1 MAC: 
  Port2 MAC: 
  Versions:  Current  Available  
  FW  2.30.3200  2.33.5000  
  PXE  3.4.0148  3.4.0460  
  UEFI  11.0.0022  14.6.0025  

  Status:  Update required

---------
Found 1 device(s) requiring firmware update...

Device #1: Updating FW ...  Done


Restart needed for updates to take effect.
I'd still like to know if I can turn it into a retail MT_1080120023 (assuming that's what it is).
 

izx

Active Member
Jan 17, 2016
113
39
28
37
If you got FRU 00D9692, its equivalent to the Mellanox MCX312A-XCBT



If it's an A5, Mlnx PSID should be MT_1080120023 (for rev. A2-A6). Verify mlxburn -d <device> -vpd gives you the same revision number.
 

Stereodude

Active Member
Feb 21, 2016
412
65
28
USA
If it's an A5, Mlnx PSID should be MT_1080120023 (for rev. A2-A6). Verify mlxburn -d <device> -vpd gives you the same revision number.
Here's what I get (less the serial number which I've edited out):
Code:
C:\Program Files\Mellanox\WinMFT>mlxburn -d mt4099_pciconf0 -vpd

  VPD-KEYWORD  DESCRIPTION  VALUE
  -----------  -----------  -----
Read Only section:

  PN  Part Number  00D9691
  EC  Revision  N32186C
  SN  Serial Number 
  V0  Misc Info  Mellanox ConnectX3 10 GbE Adapter for IBM System x
  RV  Checksum Complement  0x18
  IDTAG  Board Id  Mellanox CX3 Ethernet Adapter
  MN  Manufacture ID  A5
 

izx

Active Member
Jan 17, 2016
113
39
28
37
Ok, it's the last line. Please check that the recovery jumper is present on your card. It should be labeled FLSH PRSNT or similar. Can be straight or angled. A few cards won't have pins but the holes in the PCB should be there.
 

Stereodude

Active Member
Feb 21, 2016
412
65
28
USA
Ok, it's the last line. Please check that the recovery jumper is present on your card. It should be labeled FLSH PRSNT or similar. Can be straight or angled. A few cards won't have pins but the holes in the PCB should be there.
The through holes are on the PCB, but there is no header for a jumper. I can solder one on pretty easily.
 

izx

Active Member
Jan 17, 2016
113
39
28
37
So if I solder a 2 pin header and close the jumper what's that get me, or what's the next step?
You don't need to solder header pins there, just use a little bare copper wire (solid or multistranded) to loop through the vias and give it a few twists to short them. Now plug the card back in -- it should boot without the firmware in a "DFU" mode. This is just to check you have a means to recover the card if somehow it gets bricked.

(Thanks @Stereodude for this info:) See what it shows up as in MST and Device Manager -- should show up as a Memory Controller in Linux, or a PCI Memory Controller on Windows. Run mststatus to verify it sees the "livefished" card as a Mellanox PCI conf interface (mtxxx_pciconf0). On Linux, lspci -vv should also provide further information, in the form of Mellanox (Flash Recovery Mode) or similar.

Here's the Mellanox docs on livefish: Dropbox - Mellanox_MFT_Livefish.pdf
 

Stereodude

Active Member
Feb 21, 2016
412
65
28
USA
Okay, so the card be recovered if bricked with the vias shorted and using mlxburn?

But how do I get the IBM card to take the standard Mellanox firmware in the first place or is livefishing the card and using mlxburn how I get the standard firmware on the card?
 

izx

Active Member
Jan 17, 2016
113
39
28
37
Cross-flashing to Mellanox FW (ConnextX-2 or ConnextX-3 based devices only)
Not tested with all devices - may brick yours permanently - YMMV!

  • First, backup the existing configuration if you ever need to recover it and to compare notes post-flash
1. Save basic info such as GUIDs, MACs, etc.:
Code:
flint -d <device> query full > flint_query.txt
2. Save low-level flash chip info:
Code:
flint -d <device> hw query > flint_hwinfo.txt
3. Save existing FW:
Code:
flint -d <device> ri orig_firmware.bin
4. Save existing FW configuration:
Code:
flint -d <device> dc orig_firmware.ini
5. Save existing PXE ROM image (if any):
Code:
flint -d <device> rrom orig_rom.bin
6. Save existing PCI VPD (vital product data):
Code:
mlxburn -d <device> -vpd > orig_vpd.txt
  • Now, download "custom" Mellanox firmware tarball for your adapter from here (e.g. for the MCX312A-XCBT) and extract. Note that it must contain a multi-adapter MLX firmware file, NOT a BIN file.
  • Finally, customize and burn:
1. Identify the correct configuration file for your device/revision (here, it's MCX_312A-XCB_A2-A6.ini)
2. "Burn" the new firmware to an image file (not the device) with the correct config:
Code:
mlxburn -fw fw-ConnectX3-rel.mlx -conf MCX312A-XCB_A2-A6.ini -wrimage mlnx_firmware.bin
3. Verify the image is good with flint (no errors or warnings, should say "is bootable"):
Code:
flint -i mlnx_firmware.bin verify
4. And check that the image's FW version and PSID are what you want:
Code:
flint -i mlnx_firmware.bin query full
5. Let's roll!
Code:
flint -d <device> -i mlnx_firmware.bin -allow_psid_change burn
Reboot, and verify with flint that the PSID has changed while the GUIDs/MACs are the same as before.
 
Last edited:

izx

Active Member
Jan 17, 2016
113
39
28
37
Okay, so the card be recovered if bricked with the vias shorted and using mlxburn?
Yes. mlxburn uses flint to do the actual flashing, so either works.

But how do I get the IBM card to take the standard Mellanox firmware in the first place or is livefishing the card and using mlxburn how I get the standard firmware on the card?
As I laid out in the steps above, flint has an override to allow the PSID change. mlxburn includes card-specific information, such as the PSID, from the configuration file into the generic blob to generate a specific fw image. flint has tons of other overrides -- Mellanox flashing tools have always been insanely flexible. Heck, it'll let you write arbitrary data to arbitrary flash addresses if you want!

From what I saw in the MFT manual, the process appears to be more involved beginning with the ConnectX-4 series. That's why I said only CX2 & CX3 :)

Also, this disclaimer about livefish mode straight from the MFT manual:
Burning an image to a ConnectX®-3 adapter in Flash recovery mode may fail on some server types (that use PCIe spread spectrum). The tool may not be able to recognize the device’s PCI CONF0 or the image burn may not complete successfully. To burn the device, use the MTUSB-1 connection.
The MTUSB-1 is a USB-to-I2C adapter, and I'm sure Mellanox has sufficient documentation somewhere to let you use any generic USB-to-I2C. Of course, the very next disclaimer says:
When running flint/mlxburn via an MTUSB-1 device, a burn/query command may take up to 45 minutes to complete. To accelerate the burn process add the flag -no_flash_verify to the command line which skips the flash verification step. This flag, however, does not verify if the image is burnt correctly.
:D
 
  • Like
Reactions: Stereodude

Stereodude

Active Member
Feb 21, 2016
412
65
28
USA
The whole sequence worked! PSID changed, GUIDs/MACs didn't. It now has the latest firmware. (I've removed the MACs)
Code:
Image type:          FS2
FW Version:          2.36.5000
FW Release Date:     26.1.2016
MIC Version:         2.0.0
Config Sectors:      2
PRS Name:            cx3-1_MCX312A.prs
Rom Info:            type=PXE version=3.4.460 devid=4099
                     type=UEFI version=14.6.25
Device ID:           4099
Description:         Node             Port1            Port2            Sys image
GUIDs:               ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
MACs:
VSD:               
PSID:                MT_1080120023
Interestingly it still has some IBM traces in it though. The VPD didn't change for example.
Code:
VPD-KEYWORD  DESCRIPTION  VALUE
  -----------  -----------  -----
Read Only section:

  PN  Part Number  00D9691
  EC  Revision  N32186C
  SN  Serial Number  
  V0  Misc Info  Mellanox ConnectX3 10 GbE Adapter for IBM System x
  RV  Checksum Complement  0x18
  IDTAG  Board Id  Mellanox CX3 Ethernet Adapter
  MN  Manufacture ID  A5
6. Save existing PCI VPD (vital product data):
Code:
mlxburn -d <device> -vpd > orig_vpd.txt
Reboot, and verify with flint that the PSID has changed while the GUIDs/MACs are the same as before.
You should probably put a note here for the vpd saving to use the pciconfX device, not pci_crX. I ended up with a orig_vpd.txt that just had an error message in it.
 

izx

Active Member
Jan 17, 2016
113
39
28
37
The whole sequence worked! PSID changed, GUIDs/MACs didn't. It now has the latest firmware.
Very nice! Thanks for being the guinea pig :)

You should probably put a note here for the vpd saving to use the pciconfX device, not pci_crX. I ended up with a orig_vpd.txt that just had an error message in it.
I will. I'd thought about putting a note to not use the cr0 interface because it's s.l.o.w. compared to direct PCI, and this is another reason not to!
 

izx

Active Member
Jan 17, 2016
113
39
28
37
Interestingly it still has some IBM traces in it though. The VPD didn't change for example.
Code:
VPD-KEYWORD  DESCRIPTION  VALUE
  -----------  -----------  -----
Read Only section:

  PN  Part Number  00D9691
  EC  Revision  N32186C
  SN  Serial Number 
  V0  Misc Info  Mellanox ConnectX3 10 GbE Adapter for IBM System x
  RV  Checksum Complement  0x18
  IDTAG  Board Id  Mellanox CX3 Ethernet Adapter
  MN  Manufacture ID  A5
Yeah, the VPD shouldn't (it's supposed to be read only). I notices the VPD "row" headers were rewritten: row 2 now reads Revision, but that data remains in row 6 as before.

Mlxburn will let you overwrite the VPD data, but those values aren't in the firmware or conf files. You'd need someone with a Mellanox "original" to tell you what they should be set to. It's just dressing anyway -- doesn't affect operation.

The card does identify as Mellanox in Device Manager, right?
 

Lance Joseph

Member
Oct 5, 2014
79
32
18
Cross-flashing to Mellanox FW (ConnextX-2 or ConnextX-3 based devices only)
Not tested with all devices - may brick yours permanently - YMMV!
This was a very helpful post, @izx !! Thank you.
I've successfully flashed one MCX313A and two MCX314A (ConnectX-3 QSFP+) cards.

Just a heads up for anyone else that comes along.
If you'd like to switch between Infiniband/Ethernet/VPI modes, just use the following command:
mst.exe status
This will show you the device name. In my case, the device name was "mt4099_pciconf0"
To set a given port to operate in Infiniband mode, I used the following command:
mlxconfig.exe -d mt4099_pciconf0 set LINK_TYPE_P2=1
This would set port 2 to Infiniband mode.

Cheers
 

Kuz

New Member
Oct 7, 2016
22
16
3
41
I am trying to do this as well with a MCX314A but it looks like Mellanox has removed the custom firmware page from their site. Anyone able to share another source for that mcx314a?

Thanks!

**FOUND it** for anyone else looking they still have it up on their korean site ;)

http://kr.mellanox.com/page/custom_firmware_table
 
Last edited:

kiteboarder

Active Member
May 10, 2016
102
39
28
41
Here is what I did after I could not find the custom firmware files. This worked for a Dell branded card. Note the below instructions do not include the steps of using a shorting wire to short the holes on the card. See the info in the posts above for those details. Just remember to remove the short after you update the fw.

Step 1 - download and install
MLNX_firmware_tool_WinMFT_x64_4_7_0_42.exe

Step 2 - download:
fw-ConnectX3-rel-2_40_7000-MCX312A-XCB_A2-A6-FlexBoot-3.4.746.bin.zip
Extract .bin file and rename in to fw.bin
copy it to:
C:\Program Files\Mellanox\WinMFT>

Step 3 - open command prompt as administrator
cd to directory:
C:\Program Files\Mellanox\WinMFT>

Run:
0) mst status
1) mlxfwmanager
2) flint -d mt26448_pci_cr0 -i fw.bin burn -allow_psid_change

where "mt26448_pci_cr0" is the name output from commands 0 or 1

The -allow_psid_change switch is dangerous and I make no guarantees that this will not brick your card. YMMV!!! But it worked perfectly for my Dell branded card.

Good luck.