Flash OEM (IBM) Mellanox ConnectX-3 EN with stock firmware?

[Stereodude]

So I picked up an IBM flavored Mellanox ConnectX-3 EN (MCX312A-XCBT / Dual 10GbE SFP+) from eBay. It's FRU: 00D9692. The latest WinOF driver installed without issue (Windows 7 Pro).

The card has a PSID of IBM1080111023 so the standard MT_1080110023 firmware won't load on it. Here's what I see:

------------------------------------------------------------------------
C:\Program Files\Mellanox\WinMFT>flint -d mt4099_pci_cr0 -i fw-ConnectX3-rel-2_34_5000-MCX312A-XCB_A1-FlexBoot-3.4.521.bin burn

Current FW version on flash: 2.30.3200
New FW version: 2.34.5000

-E- PSID mismatch. The PSID on flash (IBM1080111023) differs from the PSID in the given image (MT_1080110023).
------------------------------------------------------------------------

Is there a way to flash the generic firmware? I saw this guide but I'm not sure if I can apply it to my case or not. The firmware I have is a .bin, not a .mlx, and I'm not sure about where to get an equivalent .ini file.

Has anyone tried this or know how it can be done?

 

[Stereodude]

I guess I should have looked a little harder. I found this: http://www.mellanox.com/page/custom_firmware_table

However, the ConnectX-3 EN download package is for rel-2.36.5000, and the firmware page shows rel-2.34.5000 for the MT_1080110023. I'm not sure whether the IBM1080111023 is a MT_1080110023 or a MT_1080120023...

The sticker on the back of the card says Rev. A5 and was made in 2014-03-27 which would seem to suggest it might be equivalent to a MT_1080120023, but I'm not sure.

Since I don't have my twinax cable yet or anything to plug this into, I guess I will sit tight for a while. Hopefully someone can shed some light on this.

 

[Stereodude]

So someone on another forum pointed out the mlxup utility for Mellanox cards. That brought the card up from 2.30.3200 to 2.33.5000. Here's what I saw (MAC addresses removed by me):

Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:  ConnectX3
  Part Number:  00D9691_Ax
  Description:  Mellanox ConnectX-3 EN Dual-port SFP+ 10GbE Adapter
  PSID:  IBM1080111023
  PCI Device Name:  mt4099_pci_cr0
  Port1 MAC: 
  Port2 MAC: 
  Versions:  Current  Available  
  FW  2.30.3200  2.33.5000  
  PXE  3.4.0148  3.4.0460  
  UEFI  11.0.0022  14.6.0025  

  Status:  Update required

---------
Found 1 device(s) requiring firmware update...

Device #1: Updating FW ...  Done


Restart needed for updates to take effect.

I'd still like to know if I can turn it into a retail MT_1080120023 (assuming that's what it is).

 

[izx]

If you got FRU 00D9692, its equivalent to the Mellanox MCX312A-XCBT

If it's an A5, Mlnx PSID should be MT_1080120023 (for rev. A2-A6). Verify mlxburn -d <device> -vpd gives you the same revision number.

 

[Stereodude]

If it's an A5, Mlnx PSID should be MT_1080120023 (for rev. A2-A6). Verify mlxburn -d <device> -vpd gives you the same revision number.

Here's what I get (less the serial number which I've edited out):

C:\Program Files\Mellanox\WinMFT>mlxburn -d mt4099_pciconf0 -vpd

  VPD-KEYWORD  DESCRIPTION  VALUE
  -----------  -----------  -----
Read Only section:

  PN  Part Number  00D9691
  EC  Revision  N32186C
  SN  Serial Number 
  V0  Misc Info  Mellanox ConnectX3 10 GbE Adapter for IBM System x
  RV  Checksum Complement  0x18
  IDTAG  Board Id  Mellanox CX3 Ethernet Adapter
  MN  Manufacture ID  A5

 

[izx]

Ok, it's the last line. Please check that the recovery jumper is present on your card. It should be labeled FLSH PRSNT or similar. Can be straight or angled. A few cards won't have pins but the holes in the PCB should be there.

 

[Stereodude]

Ok, it's the last line. Please check that the recovery jumper is present on your card. It should be labeled FLSH PRSNT or similar. Can be straight or angled. A few cards won't have pins but the holes in the PCB should be there.

The through holes are on the PCB, but there is no header for a jumper. I can solder one on pretty easily.

 

[Stereodude]

So if I solder a 2 pin header and close the jumper what's that get me, or what's the next step?

 

[izx]

So if I solder a 2 pin header and close the jumper what's that get me, or what's the next step?

You don't need to solder header pins there, just use a little bare copper wire (solid or multistranded) to loop through the vias and give it a few twists to short them. Now plug the card back in -- it should boot without the firmware in a "DFU" mode. This is just to check you have a means to recover the card if somehow it gets bricked.

(Thanks @Stereodude for this info See what it shows up as in MST and Device Manager -- should show up as a Memory Controller in Linux, or a PCI Memory Controller on Windows. Run mststatus to verify it sees the "livefished" card as a Mellanox PCI conf interface (mtxxx_pciconf0). On Linux, lspci -vv should also provide further information, in the form of Mellanox (Flash Recovery Mode) or similar.

Here's the Mellanox docs on livefish: Dropbox - Mellanox_MFT_Livefish.pdf

 

[Stereodude]

Okay, so the card be recovered if bricked with the vias shorted and using mlxburn?

But how do I get the IBM card to take the standard Mellanox firmware in the first place or is livefishing the card and using mlxburn how I get the standard firmware on the card?

 

[izx]

Cross-flashing to Mellanox FW (ConnextX-2 or ConnextX-3 based devices only)
Not tested with all devices - may brick yours permanently - YMMV!

• First, backup the existing configuration if you ever need to recover it and to compare notes post-flash

1. Save basic info such as GUIDs, MACs, etc.:

flint -d <device> query full > flint_query.txt

2. Save low-level flash chip info:

flint -d <device> hw query > flint_hwinfo.txt

3. Save existing FW:

flint -d <device> ri orig_firmware.bin

4. Save existing FW configuration:

flint -d <device> dc orig_firmware.ini

5. Save existing PXE ROM image (if any):

flint -d <device> rrom orig_rom.bin

6. Save existing PCI VPD (vital product data):

mlxburn -d <device> -vpd > orig_vpd.txt

• Now, download "custom" Mellanox firmware tarball for your adapter from here (e.g. for the MCX312A-XCBT) and extract. Note that it must contain a multi-adapter MLX firmware file, NOT a BIN file.
  
• Finally, customize and burn:

1. Identify the correct configuration file for your device/revision (here, it's MCX_312A-XCB_A2-A6.ini)
2. "Burn" the new firmware to an image file (not the device) with the correct config:

mlxburn -fw fw-ConnectX3-rel.mlx -conf MCX312A-XCB_A2-A6.ini -wrimage mlnx_firmware.bin

3. Verify the image is good with flint (no errors or warnings, should say "is bootable"):

flint -i mlnx_firmware.bin verify

4. And check that the image's FW version and PSID are what you want:

flint -i mlnx_firmware.bin query full

5. Let's roll!

flint -d <device> -i mlnx_firmware.bin -allow_psid_change burn

Reboot, and verify with flint that the PSID has changed while the GUIDs/MACs are the same as before.

 

[izx]

Okay, so the card be recovered if bricked with the vias shorted and using mlxburn?

Yes. mlxburn uses flint to do the actual flashing, so either works.

But how do I get the IBM card to take the standard Mellanox firmware in the first place or is livefishing the card and using mlxburn how I get the standard firmware on the card?

As I laid out in the steps above, flint has an override to allow the PSID change. mlxburn includes card-specific information, such as the PSID, from the configuration file into the generic blob to generate a specific fw image. flint has tons of other overrides -- Mellanox flashing tools have always been insanely flexible. Heck, it'll let you write arbitrary data to arbitrary flash addresses if you want!

From what I saw in the MFT manual, the process appears to be more involved beginning with the ConnectX-4 series. That's why I said only CX2 & CX3

Also, this disclaimer about livefish mode straight from the MFT manual:

Burning an image to a ConnectX®-3 adapter in Flash recovery mode may fail on some server types (that use PCIe spread spectrum). The tool may not be able to recognize the device’s PCI CONF0 or the image burn may not complete successfully. To burn the device, use the MTUSB-1 connection.

The MTUSB-1 is a USB-to-I2C adapter, and I'm sure Mellanox has sufficient documentation somewhere to let you use any generic USB-to-I2C. Of course, the very next disclaimer says:

When running flint/mlxburn via an MTUSB-1 device, a burn/query command may take up to 45 minutes to complete. To accelerate the burn process add the flag -no_flash_verify to the command line which skips the flash verification step. This flag, however, does not verify if the image is burnt correctly.

 

[Stereodude]

Thanks for the very detailed reply. I will try this later today.

 

[Stereodude]

The whole sequence worked! PSID changed, GUIDs/MACs didn't. It now has the latest firmware. (I've removed the MACs)

Image type:          FS2
FW Version:          2.36.5000
FW Release Date:     26.1.2016
MIC Version:         2.0.0
Config Sectors:      2
PRS Name:            cx3-1_MCX312A.prs
Rom Info:            type=PXE version=3.4.460 devid=4099
                     type=UEFI version=14.6.25
Device ID:           4099
Description:         Node             Port1            Port2            Sys image
GUIDs:               ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
MACs:
VSD:               
PSID:                MT_1080120023

Interestingly it still has some IBM traces in it though. The VPD didn't change for example.

VPD-KEYWORD  DESCRIPTION  VALUE
  -----------  -----------  -----
Read Only section:

  PN  Part Number  00D9691
  EC  Revision  N32186C
  SN  Serial Number  
  V0  Misc Info  Mellanox ConnectX3 10 GbE Adapter for IBM System x
  RV  Checksum Complement  0x18
  IDTAG  Board Id  Mellanox CX3 Ethernet Adapter
  MN  Manufacture ID  A5

6. Save existing PCI VPD (vital product data):

mlxburn -d <device> -vpd > orig_vpd.txt

Reboot, and verify with flint that the PSID has changed while the GUIDs/MACs are the same as before.

You should probably put a note here for the vpd saving to use the pciconfX device, not pci_crX. I ended up with a orig_vpd.txt that just had an error message in it.

 

[izx]

The whole sequence worked! PSID changed, GUIDs/MACs didn't. It now has the latest firmware.

Very nice! Thanks for being the guinea pig

You should probably put a note here for the vpd saving to use the pciconfX device, not pci_crX. I ended up with a orig_vpd.txt that just had an error message in it.

I will. I'd thought about putting a note to not use the cr0 interface because it's s.l.o.w. compared to direct PCI, and this is another reason not to!

 

[izx]

Interestingly it still has some IBM traces in it though. The VPD didn't change for example.

VPD-KEYWORD  DESCRIPTION  VALUE
  -----------  -----------  -----
Read Only section:

  PN  Part Number  00D9691
  EC  Revision  N32186C
  SN  Serial Number 
  V0  Misc Info  Mellanox ConnectX3 10 GbE Adapter for IBM System x
  RV  Checksum Complement  0x18
  IDTAG  Board Id  Mellanox CX3 Ethernet Adapter
  MN  Manufacture ID  A5

Yeah, the VPD shouldn't (it's supposed to be read only). I notices the VPD "row" headers were rewritten: row 2 now reads Revision, but that data remains in row 6 as before.

Mlxburn will let you overwrite the VPD data, but those values aren't in the firmware or conf files. You'd need someone with a Mellanox "original" to tell you what they should be set to. It's just dressing anyway -- doesn't affect operation.

The card does identify as Mellanox in Device Manager, right?

 

[Lance Joseph]

Cross-flashing to Mellanox FW (ConnextX-2 or ConnextX-3 based devices only)
Not tested with all devices - may brick yours permanently - YMMV!

This was a very helpful post, @izx !! Thank you.
I've successfully flashed one MCX313A and two MCX314A (ConnectX-3 QSFP+) cards.

Just a heads up for anyone else that comes along.
If you'd like to switch between Infiniband/Ethernet/VPI modes, just use the following command:
mst.exe status
This will show you the device name. In my case, the device name was "mt4099_pciconf0"
To set a given port to operate in Infiniband mode, I used the following command:
mlxconfig.exe -d mt4099_pciconf0 set LINK_TYPE_P2=1
This would set port 2 to Infiniband mode.

Cheers

 

[Kuz]

I am trying to do this as well with a MCX314A but it looks like Mellanox has removed the custom firmware page from their site. Anyone able to share another source for that mcx314a?

Thanks!

**FOUND it** for anyone else looking they still have it up on their korean site

http://kr.mellanox.com/page/custom_firmware_table

 

[kiteboarder]

Here is what I did after I could not find the custom firmware files. This worked for a Dell branded card. Note the below instructions do not include the steps of using a shorting wire to short the holes on the card. See the info in the posts above for those details. Just remember to remove the short after you update the fw.

Step 1 - download and install
MLNX_firmware_tool_WinMFT_x64_4_7_0_42.exe

Step 2 - download:
fw-ConnectX3-rel-2_40_7000-MCX312A-XCB_A2-A6-FlexBoot-3.4.746.bin.zip
Extract .bin file and rename in to fw.bin
copy it to:
C:\Program Files\Mellanox\WinMFT>

Step 3 - open command prompt as administrator
cd to directory:
C:\Program Files\Mellanox\WinMFT>

Run:
0) mst status
1) mlxfwmanager
2) flint -d mt26448_pci_cr0 -i fw.bin burn -allow_psid_change

where "mt26448_pci_cr0" is the name output from commands 0 or 1

The -allow_psid_change switch is dangerous and I make no guarantees that this will not brick your card. YMMV!!! But it worked perfectly for my Dell branded card.

Good luck.

 

[_alex]

Has anyone managed to burn a pro fw to a non-pro device yet ?
i.e. replace PSID MT_1090120019 with FW for 354a-fcct / PSID MT_1090111019