Lenovo Connectx-4 card flash to Mellanox

[AK360]

Hello helpful people,

I have just received 3x Lenovo 01GR250 CX4 2x25Gb cards. Is there an idiots guide out there to flash these to MCX4121A-ACAT firmware which is what they are under the skin ??

I searched the forums but couldn't see anything for Connectx 4 other than someone trying to hack a 50Gb to 100Gb.

Any help gratefully received!

Alex

 

[i386]

I used the same procedure to flash a hpe card (825110-B21) to a vanilla mellanox nic (MCX455A-ECAT) as with the connect-x3 cards.

 

[AK360]

I used the same procedure to flash a hpe card (825110-B21) to a vanilla mellanox nic (MCX455A-ECAT) as with the connect-x3 cards.

Thank you It did work with a bit of modifying, for any one wondering:

********** Backup Settings **************
flint -d mt4117_pciconf0 query full > flint_query.txt

flint -d mt4117_pciconf0 hw query > flint_hwinfo.txt

flint -d mt4117_pciconf0 ri orig_firmware.bin

flint -d mt4117_pciconf0 dc orig_firmware.ini

mlxburn -d mt4117_pciconf0 -vpd > orig_vpd.txt

************** Write Mellanox Firmware ***************

flint -d mt4117_pciconf0 -ocr hw set Flash0.WriteProtected=Disabled

flint -nofs --use_image_ps --ignore_dev_data -allow_psid_change -d mt4117_pciconf0 -i fw-ConnectX4Lx-rel-14_30_1004-MCX4121A-ACA_Ax-UEFI-14.23.17-FlexBoot-3.6.301.bin burn

************* After power power cycle program GUID and MAC **************

flint -d mt4117_pciconf2 -guid XXXX -mac XXXX -ocr sg

 

[Karpo]

Hi,

bought Connectx-4 100G Dell model:
Device Type: ConnectX4
Part Number: 06W1HY_0JJN39_Ax
Description: Mellanox ConnectX-4 Single Port EDR PCIE Adapter LP
PSID: DEL2180110032
PCI Device Name: mt4115_pciconf0

As far as I understand this is same as CX455A-ECAT.

I changed it IB->Ethernet successfully and can get 94Gb/sec in iPerf. Enabled SR-IOV in firmware, updated EXSi drivers and enabled SR-IOV in drivers but card stays in 'Reboot to apply' in ESXi 7.

Guessing that Dell firmware could be the reason so tried above steps to flash it as Mellanox CX455A-ECAT.

Not sure if this response is expected when disabling write protection:

sudo flint -d /dev/mst/mt4115_pciconf0 -ocr hw set Flash0.WriteProtected=Disabled

-W- Firmware flash cache access is enabled. Running in this mode may cause the firmware to hang.
-E- Unknown option "set" for the "Hw" command. you can use query.

Checking write protection, it seems to be still on:

sudo flint -d 03:00.0 -ocr hw query
-W- Firmware flash cache access is enabled. Running in this mode may cause the firmware to hang.
HW Info:
HwDevId 521
HwRevId 0x0
Flash Info:
Type W25QxxBV
TotalSize 0x1000000
Banks 0x1
SectorSize 0x1000
WriteBlockSize 0x10
CmdSet 0x80
QuadEn 1
Flash0.WriteProtected Top,8-SubSectors
JEDEC_ID 0x1840ef

Flashing fails:

sudo flint -nofs --use_image_ps --ignore_dev_data -allow_psid_change -d /dev/mst/mt4115_pciconf0 -i ./fw-ConnectX4-rel-12_28_2006-MCX455A-FCA_Ax-UEFI-14.21.17-FlexBoot-3.6.102.bin burn

Current FW version on flash: 12.28.4512
New FW version: 12.28.2006

Note: The new FW version is older than the current FW version on flash.

Do you want to continue ? (y/n) [n] : y
You are about to replace current PSID on flash - "DEL2180110032" with a different PSID - "MT_2160111021".
Note: It is highly recommended not to change the PSID.

Do you want to continue ? (y/n) [n] : y

Burn process will not be failsafe. No checks will be performed.
ALL flash, including the device data sections will be overwritten.
If this process fails, computer may remain in an inoperable state.

Do you want to continue ? (y/n) [n] : y

-E- Burning FS3 image failed: Cannot burn device data sections, Flash is write protected.


Any ideas on how to disable write protection?

 

[Karpo]

Answering myself, got flashing working by installing mstflint (and getting right CX455-ECA firmware, rather than FCA above):

sudo apt install mstflint
sudo mstflint -d 03:00.0 -i fw-ConnectX4-rel-12_28_2006-MCX455A-ECA_Ax-UEFI-14.21.17-FlexBoot-3.6.102.bin -allow_psid_change burn

<reboot>

sudo flint -d /dev/mst/mt4115_pciconf0 query full
Image type: FS3
FW Version: 12.28.2006
FW Release Date: 15.9.2020
Part Number: MCX455A-ECA_Ax
Description: ConnectX-4 VPI adapter card; EDR IB (100Gb/s) and 100GbE; single-port QSFP28; PCIe3.0 x16; ROHS R6
Product Version: 12.28.2006
Rom Info: type=UEFI version=14.21.17 cpu=AMD64,AARCH64
type=PXE version=3.6.102 cpu=AMD64
Description: UID GuidsNumber
Base GUID: 248a070300a8---- 4
Base MAC: 248a07a8---- 4
Image VSD: N/A
Device VSD: N/A
PSID: MT_2180110032
Security Attributes: N/A
Default Update Method: fw_ctrl

However EXSi still keeps SR-IOV as 'reboot needed', maybe Dell T5810 does not support it in bios.

 

[mcdaguo]

Hi,

bought Connectx-4 100G Dell model:
Device Type: ConnectX4
Part Number: 06W1HY_0JJN39_Ax
Description: Mellanox ConnectX-4 Single Port EDR PCIE Adapter LP
PSID: DEL2180110032
PCI Device Name: mt4115_pciconf0

As far as I understand this is same as CX455A-ECAT.

I changed it IB->Ethernet successfully and can get 94Gb/sec in iPerf. Enabled SR-IOV in firmware, updated EXSi drivers and enabled SR-IOV in drivers but card stays in 'Reboot to apply' in ESXi 7.

Guessing that Dell firmware could be the reason so tried above steps to flash it as Mellanox CX455A-ECAT.

Not sure if this response is expected when disabling write protection:

sudo flint -d /dev/mst/mt4115_pciconf0 -ocr hw set Flash0.WriteProtected=Disabled

-W- Firmware flash cache access is enabled. Running in this mode may cause the firmware to hang.
-E- Unknown option "set" for the "Hw" command. you can use query.

Checking write protection, it seems to be still on:

sudo flint -d 03:00.0 -ocr hw query
-W- Firmware flash cache access is enabled. Running in this mode may cause the firmware to hang.
HW Info:
HwDevId 521
HwRevId 0x0
Flash Info:
Type W25QxxBV
TotalSize 0x1000000
Banks 0x1
SectorSize 0x1000
WriteBlockSize 0x10
CmdSet 0x80
QuadEn 1
Flash0.WriteProtected Top,8-SubSectors
JEDEC_ID 0x1840ef

Flashing fails:

sudo flint -nofs --use_image_ps --ignore_dev_data -allow_psid_change -d /dev/mst/mt4115_pciconf0 -i ./fw-ConnectX4-rel-12_28_2006-MCX455A-FCA_Ax-UEFI-14.21.17-FlexBoot-3.6.102.bin burn

Current FW version on flash: 12.28.4512
New FW version: 12.28.2006

Note: The new FW version is older than the current FW version on flash.

Do you want to continue ? (y/n) [n] : y
You are about to replace current PSID on flash - "DEL2180110032" with a different PSID - "MT_2160111021".
Note: It is highly recommended not to change the PSID.

Do you want to continue ? (y/n) [n] : y

Burn process will not be failsafe. No checks will be performed.
ALL flash, including the device data sections will be overwritten.
If this process fails, computer may remain in an inoperable state.

Do you want to continue ? (y/n) [n] : y

-E- Burning FS3 image failed: Cannot burn device data sections, Flash is write protected.


Any ideas on how to disable write protection?

Do you know how to change Part Number: 06W1HY_0JJN39_Ax to Mellanox MCX455-ECAT ?

 

[Karpo]

After flashing with mstflint using Mellanox firmware with below switches, it also changes Part Number:
sudo mstflint -d 03:00.0 -i fw-ConnectX4-rel-12_28_2006-MCX455A-ECA_Ax-UEFI-14.21.17-FlexBoot-3.6.102.bin -allow_psid_change burn

 

[mcdaguo]

After flashing with mstflint using Mellanox firmware with below switches, it also changes Part Number:
sudo mstflint -d 03:00.0 -i fw-ConnectX4-rel-12_28_2006-MCX455A-ECA_Ax-UEFI-14.21.17-FlexBoot-3.6.102.bin -allow_psid_change burn

Although you can use flint -d mt4115_pciconf0 q full to see that the Part Number has been changed to MCX455-ECA, but you can still see the Part Number and serial number before the OEM network card in the network card information. How can I modify the PN and SN of the OEM network card information?

 

[Freebsd1976]

Although you can use flint -d mt4115_pciconf0 q full to see that the Part Number has been changed to MCX455-ECA, but you can still see the Part Number and serial number before the OEM network card in the network card information. How can I modify the PN and SN of the OEM network card information? View attachment 27809
View attachment 27808

These call vpd, vpd is readonly. it can’t be modify.
and more important, shouldn’t modify them, Even if someone could( almost impossible).

unless someone want to make “genuine” nic , just like many intel x520 on second hand market.

 

[mach3.2]

Hi,

bought Connectx-4 100G Dell model:
Device Type: ConnectX4
Part Number: 06W1HY_0JJN39_Ax
Description: Mellanox ConnectX-4 Single Port EDR PCIE Adapter LP
PSID: DEL2180110032
PCI Device Name: mt4115_pciconf0

As far as I understand this is same as CX455A-ECAT.

I changed it IB->Ethernet successfully and can get 94Gb/sec in iPerf. Enabled SR-IOV in firmware, updated EXSi drivers and enabled SR-IOV in drivers but card stays in 'Reboot to apply' in ESXi 7.

Guessing that Dell firmware could be the reason so tried above steps to flash it as Mellanox CX455A-ECAT.

Not sure if this response is expected when disabling write protection:

sudo flint -d /dev/mst/mt4115_pciconf0 -ocr hw set Flash0.WriteProtected=Disabled

-W- Firmware flash cache access is enabled. Running in this mode may cause the firmware to hang.
-E- Unknown option "set" for the "Hw" command. you can use query.

Checking write protection, it seems to be still on:

sudo flint -d 03:00.0 -ocr hw query
-W- Firmware flash cache access is enabled. Running in this mode may cause the firmware to hang.
HW Info:
HwDevId 521
HwRevId 0x0
Flash Info:
Type W25QxxBV
TotalSize 0x1000000
Banks 0x1
SectorSize 0x1000
WriteBlockSize 0x10
CmdSet 0x80
QuadEn 1
Flash0.WriteProtected Top,8-SubSectors
JEDEC_ID 0x1840ef

Flashing fails:

sudo flint -nofs --use_image_ps --ignore_dev_data -allow_psid_change -d /dev/mst/mt4115_pciconf0 -i ./fw-ConnectX4-rel-12_28_2006-MCX455A-FCA_Ax-UEFI-14.21.17-FlexBoot-3.6.102.bin burn

Current FW version on flash: 12.28.4512
New FW version: 12.28.2006

Note: The new FW version is older than the current FW version on flash.

Do you want to continue ? (y/n) [n] : y
You are about to replace current PSID on flash - "DEL2180110032" with a different PSID - "MT_2160111021".
Note: It is highly recommended not to change the PSID.

Do you want to continue ? (y/n) [n] : y

Burn process will not be failsafe. No checks will be performed.
ALL flash, including the device data sections will be overwritten.
If this process fails, computer may remain in an inoperable state.

Do you want to continue ? (y/n) [n] : y

-E- Burning FS3 image failed: Cannot burn device data sections, Flash is write protected.


Any ideas on how to disable write protection?

For those who landed here using Google, you need to use the OEM version of MFT to disable flash write protection. The regular flint won't allow you to disable flash write protection.

You do that by installing MFT with the --oem flag.

See step c under Linux install.

Compilation and Installation

docs.nvidia.com

 

[obovata]

Thank you everyone for the helpful posts in this thread. I picked up a Dell branded Connectx-4 LX (p/n 0MRT0D) recently and was able to successfully flash to the stock Mellanox firmware (14.32.1010, PSID MT_2420110034).

Unfortunately the Mellanox firmware produces an issue with the link LEDs on the card. They remain illuminated green even when there were no transceivers installed. The activity LEDs still worked correctly. Everything else on the card seemed to work normally but I was left wondering if there were any other not so obvious issues from the firmware change.

I ended up flashing back to the original Dell firmware. Thank you AK360 for including that flint command to back up the original firmware! I can confirm that restoring from it works. The LEDs on my card are now back to operating as they should. I also noticed that I did not have to manually set the GUID and MAC again after flashing despite the Mellanox to Dell PSID change.

Curiously the latest Dell firmware is a bit newer than the latest Mellanox one (14.32.2004 vs. 14.32.1010). Unfortunately the release notes Dell provides leaves the feature summary and fixes sections blank so who knows what difference it makes. I did catch that the UEFI driver was incremented up from 14.25.17 to 14.25.18 as well:



If anyone with a Dell branded Connectx-4 LX that needs to go back to the original Dell firmware and did not make a backup, I found that you can extract out the firmware from the Windows installer on Dell's site. Use 7zip on the exe and you'll find the .bin in the payload folder. There are few but Dell helpfully includes the applicable part numbers in the filename: fw-ConnectX4Lx-rel-14_32_2004-020NJD_0MRT0D_Ax-UEFI-14.25.18-FlexBoot-3.6.502.bin.