Crossflash HPE 640sfp28 to MCX4121A-ACAT

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

vangoose

Active Member
May 21, 2019
326
106
43
Canada
Followed all the instructions to flash OEM cx3 to MCX firmware https://forums.servethehome.com/ind...lanox-connectx-3-en-with-stock-firmware.8689/

HPE 640sfp28 = MCX4121A-ACAT
Procedures to flash CX4 is exactly same, compare PSID -
HPE - HP_2420110034
MCX - MT_2420110034

1. download and install WinMFT_x64_4_14_2_17.exe
2. download firmware fw-ConnectX4Lx-rel-14_27_1016-MCX4121A-ACA_Ax-UEFI-14.20.19-FlexBoot-3.5.901.bin.zip
3. cd to firmware download directory
4. flint -d mt4117_pciconf0 -i fw-ConnectX4Lx-rel-14_27_1016-MCX4121A-ACA_Ax-UEFI-14.20.19-FlexBoot-3.5.901.bin -allow_psid_change burn
5. I flashed it in Windows VM with pci-passthrough, so reboot ESXi is needed.


Before
Code:
mlxfwmanager.exe
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX4LX
  Part Number:      817753-B21_Ax
  Description:      HPE Ethernet 10/25Gb 2-port 640SFP28 Adapter
  PSID:             HP_2420110034
  PCI Device Name:  mt4117_pciconf0
  Base MAC:         9cdc714c2250
  Versions:         Current        Available
     FW             14.26.1040     N/A
     PXE            3.5.0803       N/A
     UEFI           14.19.0014     N/A

  Status:           No matching image found

After
Code:
mlxfwmanager.exe
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX4LX
  Part Number:      MCX4121A-ACA_Ax
  Description:      ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6
  PSID:             MT_2420110034
  PCI Device Name:  mt4117_pciconf0
  Base MAC:         9cdc714c2250
  Versions:         Current        Available
     FW             14.27.1016     N/A
     PXE            3.5.0901       N/A
     UEFI           14.20.0019     N/A

  Status:           No matching image found
 
Last edited:

vangoose

Active Member
May 21, 2019
326
106
43
Canada
After crossflash, all features are available, sr-iov is working properly on ESXi 6.7U3 and I can assign VF to FreeNAS and it sees it after loader hw.pci.honor_msi_blacklist = 0.
 
  • Like
Reactions: Exhaust8890

MountainBofh

Beating my users into submission
Mar 9, 2024
375
264
63
Granted this is an older thread, but it worked great for me. I'm providing some slightly more noob friendly instructions

1. Grab the newest MFT tools here for whatever platform you're running on. I used my Windows test machine to perform this, but I don't see why Linux wouldn't work either. NVIDIA Firmware Tools (MFT)

2. After installing the tools, run the mlxfwmanager tool (in Windows its located in c:\program files\mellanox\winmft, in Linux look in /usr/bin ). Make a note (or screenshot ) of the info it spits out, the PSID in particular. On one of my HP cards prior to cross flashing, here's what I got

Querying Mellanox devices firmware ...

Device #1:
----------

Device Type: ConnectX4LX
Part Number: 817753-B21_Ax
Description: HPE Ethernet 10/25Gb 2-port 640SFP28 Adapter
PSID: HP_2420110034
PCI Device Name: /dev/mst/mt4117_pciconf0
Base MAC: 9440c9c2f478
Versions: Current Available
FW 14.24.9000 N/A
PXE 3.5.0603 N/A
UEFI 14.17.0013 N/A

Status: No matching image found



3. Go to Nvidia's site, and grab the newest firmware. Link is Firmware for ConnectX®-4 Lx EN . For my cards, the OPN was MCX4121A-ACAT. Download the zip file, and extract the firmware to wherever you have the mlxfwmanager tool.

4. Run the following command. If there's a newer firmware out, change the filename accordingly. For Windows, I ran the following.
flint -d mt4117_pciconf0 -i fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACA_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin -allow_psid_change burn

linux note - for the above command, you'll need to run lspci to get the device address. On my linux box, I ran the following command.
flint -d 01:00.0 -i fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACA_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin -allow_psid_change burn

You'll get a prompt asking for confirmation, and output like this.

Done.ng flash section: 100%←[2K
Current FW version on flash: 14.24.9000
New FW version: 14.32.1010


You are about to replace current PSID on flash - "HP_2420110034" with a different PSID - "MT_2420110034".
Note: It is highly recommended not to change the PSID.

Do you want to continue ? (y/n) [n] : y
Burning FW image without signatures - OK
Restoring signature - OK
-I- To load new FW run mlxfwreset or reboot machine.


I rebooted my Windows test machine, and in device manager the card went from an "HP 10/25gb ethernet" to a Mellanox card, so victory! If you're running on Windows, I'd strongly recommend grabbing the newest drivers from Nvidia and use those instead of the ancient crap drivers built into Windows. You can find them here. Mellanox OFED for Windows - WinOF / WinOF-2
 
Last edited:

Thomas H

Member
Dec 2, 2017
95
77
18
54
@MountainBofh
Excellent write up! Detailed, easy to follow, and worked for me. Thanks! For #3, you can confirm which OPN by clicking an item from the list until it matches your PSID number you got from step #2. For OPN "MCX4121A-ACAT" = PSID "MT_2420110034" = from step #2 PSID: HP_2420110034
 
  • Like
Reactions: blunden

BlueChris

Active Member
Jul 18, 2021
155
59
28
53
Athens-Greece
Guys late in the party. I just bought this HPE card and i have it in my proxmox. Do i really need this crossflash? i know its a rhetorical question but the card is working fine so i don't really know what i miss...
I tried it also with passthrough in a windows VM with latest Nvidia drivers and is working fine.
 

blunden

Well-Known Member
Nov 29, 2019
809
266
63
Guys late in the party. I just bought this HPE card and i have it in my proxmox. Do i really need this crossflash? i know its a rhetorical question but the card is working fine so i don't really know what i miss...
I tried it also with passthrough in a windows VM with latest Nvidia drivers and is working fine.
Some OEM firmwares disable certain advanced features like SR-IOV, if I'm not mistaken. If yours doesn't or you don't need any potentially missing feature, feel free to stick to the OEM firmware. :)
 
  • Like
Reactions: BlueChris

BlueChris

Active Member
Jul 18, 2021
155
59
28
53
Athens-Greece
Some OEM firmwares disable certain advanced features like SR-IOV, if I'm not mistaken. If yours doesn't or you don't need any potentially missing feature, feel free to stick to the OEM firmware. :)
Thx, i will try to enable the SRVio which seems more difficult in mellanox cards than the intel cards that i am used to and if i got troubles i will flash the firmware.
 

blunden

Well-Known Member
Nov 29, 2019
809
266
63
Thx, i will try to enable the SRVio which seems more difficult in mellanox cards than the intel cards that i am used to and if i got troubles i will flash the firmware.
No need to do so if you don't need that feature. :) Also, I think SR-IOV might be hard to get working properly on the CX3 anyway.
 

d1912

New Member
Mar 7, 2024
7
3
3
Thanks for the guide. I followed MountainBofh's instructions, which seemed to work fine for my HPE Ethernet 10/25Gb 2-port 640SFP28 Adapter, exact same as theirs.

However, now I am wondering why did I mess with a working thing...

I broke my router setup because the HCA passthrough broke completely after the FW upgrade.

I passthrough the HCA (completely, both devices) to a OpenWRT VM. I am using Proxmox as the host OS. Inside my VM (which runs OpenWRT w/ 6.6 Kernel) dmesg output now shows:

Code:
mlx5_core 0000:06:10.0: 0.000 Gb/s available PCIe bandwidth (Unknown x255 link)
mlx5_core 0000:06:10.0: Port module event: module 0, Cable plugged
And so on, but I get no link at all, on either of the ports (both say 0.000 Gb/s). After some time, about 4 minutes, I will get a hw checksum failure on one port as well. Unfortunately, I don't remember if the previous boot would show this fake port information or if it was accurate (could be a red herring).

I noticed quickly that the host was seeing the board, and its dmesg output was correct (link speed etc)! But I saw now the host had the driver loading, so it wasn't getting assigned to vfio-pcie. I blacklisted mlx5_core with a fake install, to no avail. The HCA is still broken inside OpenWRT's VM.

Then, I flashed OpenWRT to a USB Drive, and booted the machine, where the HCA came up perfectly and everything works just fine. So I cut out Proxmox and run my router from a USB key while I try to gather next steps.

Does anyone know what could be wrong? I'm not trying to use SR-IOV, just literally dump the whole HCA into a VM. Before the FW upgrade, things were fine.

Is it possible the FW upgrade changed some setting that caused this path to break? I noticed the HCA has SR_IOV enabled ( SRIOV_EN = 1), maybe that was disabled before and got enabled?

Lesson learned: take backups of the firmware, and also dump all the settings so I can compare what changed.
 

d1912

New Member
Mar 7, 2024
7
3
3
Alright, I was able to save myself. I found the HPE firmware for the card at https://support.hpe.com/connect/s/s...4c2b59c36ce4642ba7da920d7&tab=revisionHistory and force flashed it back, and after a poweroff and poweron, the VM was getting an IP again!

Oddly enough, I went through the "use SR-IOV and VFs" path with the Mellanox FW to try and do things the "right way"... and while I was able to create one VF on one port, on the other port I got a write error and when I tried to use the one VF I made, I got sr-iov errors in the host (lots of them). I'm starting to think this platform (Core i7 7700K on an old ASUS motherboard) doesn't really have proper support for VT-d and SR-IOV.

Also, for anyone else in the same bind: in Proxmox, I have to use the SeaBIOS BIOS type. If I have a VM with OVFM UEFI, the passthrough HCA also fails to get an IP just like with the Mellanox FW issue.

Oh, and inside my FM I still get the weird 0.000 Gb/s available PCIe bandwidth (Unknown x255 link) message, but who cares, it works.

Also, the hardware checksum still happens, but I have no adverse effects:

Code:
[  526.200533] mlx5_core 0000:06:10.1 eth1: hw csum failure
[  526.201090] skb len=148 headroom=78 headlen=148 tailroom=30
[  526.201090] mac=(64,14) net=(78,20) trans=98
[  526.201090] shinfo(txflags=0 nr_frags=0 gso(size=0 type=0 segs=0))
[  526.201090] csum(0x5776eda9 ip_summed=2 complete_sw=0 valid=0 level=0)
[  526.201090] hash(0x1ae114c4 sw=0 l4=1) proto=0x0800 pkttype=0 iif=3
[  526.203685] dev name=eth1 feat=0x0610a1821cd14ba9
 
Last edited:
  • Like
Reactions: Djeross and TRACKER

juey90

New Member
May 13, 2024
4
1
3
Hi, i used this tutorial for flashing my hp640 nic to mcx4121a-acat, but sadly it wont let me replace the the guid and mac.
It says "N/A" after trying a few combinations (--guid b8830XXXXXXXXXXX --mac b8830XXXXXXX, --guid 0xb8830XXXXXXXXXXX --mac 0xb8830XXXXXXX). Is there any method to get this parameters set ?
 

richardm

Member
Sep 27, 2013
47
16
8
Anyone know if any variant of the CX4121 -- OEM or otherwise -- can be hacked into providing NPAR? It's the no-SRIOV-for-you consolation prize for those of us with chipset PCIe slots that lack proper ACS support. The Mellanox marketing slicks have NPAR spelled out specifically but I've spotted Lenovo and Dell docs clearly stating no NPAR for these things. So perhaps I have a good reason to cross-flash?

I've yet find a working ACS override for Windows DDA. Also I can't find anything in Mellonvidia docs that even mentions NIC partitions.
[confusedemoji]
 

kapone

Well-Known Member
May 23, 2015
1,317
780
113
PSA: After flashing stock Mellanox firmware...AND making a few tweaks on a Mac/Hack...these (or any other MCX4121A-ACAT) work OOTB. Apple added an in-house developed MLX5 driver (but it's driver-kit based) since MacOS Ventura, if I'm not wrong.

A Mac with 25gb networking... :) I'm tickled!
 

richardm

Member
Sep 27, 2013
47
16
8
Anyone know if any variant of the CX4121 -- OEM or otherwise -- can be hacked into providing NPAR?
Answering myself: Use mlxconfig to set NUM_OF_PF=4 for two fresh NPARs.

Seems to work with ESXi (which doesn't need it because SR-IOV works). Seems to NOT work with Windows where the nmlx driver interprets the additional appearances as kernel debug devices. Tried DDA passthru to a VM -- one NPAR did work in the guest; the other was again detected as kernel debug.

Janky half-feature.
 
  • Like
Reactions: blunden and nexox

kapone

Well-Known Member
May 23, 2015
1,317
780
113
PSA: After flashing stock Mellanox firmware...AND making a few tweaks on a Mac/Hack...these (or any other MCX4121A-ACAT) work OOTB. Apple added an in-house developed MLX5 driver (but it's driver-kit based) since MacOS Ventura, if I'm not wrong.

A Mac with 25gb networking... :) I'm tickled!
PSA2: As of Sequoia 15.2...it looks like ZERO tweaks are needed. They work totally OOB.
 
  • Like
Reactions: nexox

Djeross

New Member
Feb 24, 2016
15
22
3
Hi !

I cross-referenced quite a lot of posts before getting everything right, encountering some troubles along the way, so I thought it would be a good idea to share my findings.
Everything here is merely a compilation of others' work, so cheers to them first and foremost :
@MountainBofh in this same thread, of course the great @fohdeesha as usual for his reference post about ConnectX-3, @AK360 for his procedure, @up3up4 for the missing piece, and probably some I forgot — sorry.

Feel free to skip to step 7 for a complete script if you're not interested in the detailed version with explanations.
I’ve tried to be as exhaustive as possible.
This was tested on a live version of Ubuntu Server 24.04.1

Step 1: Download and Install the Tools
Some packages are needed : sudo apt -y install gcc make dkms unzip wget
Find the latest version of nVidia Firmware Tools (MFT) URL, download and unpack it :
Bash:
wget https://network.nvidia.com/products/adapter-software/firmware-tools
tar xvf mft-4.31.0-149-x86_64-deb.tgz
We need to install the OEM version of MFT on OEM cards to be able to disable flash write protection.
The regular flint wouldn't allow me to disable it. Thanks @mach3.2 for the tip, not sure why it is not in others' procedures, maybe it wasn't necessary before.
Installation is simple; just adapt the path to the version :
sudo ./<mft-xxx_folder>/install.sh --oem

Step 2: Prepare the flash and Backup
All MFT tools address the target hardware device using an mst device name. This name is assigned by running the command sudo mst start (in Windows, it is not required to run the mst start command) for PCI and I2C access.
Run sudo mst status to list the devices and their available aliases, eg. /dev/mst/mt4117_pciconf0.

Once done, we backup some settings and retrieve the original configuration & firmware for safety, as well as to retrieve data if needed (replace device name with yours) :
Bash:
sudo flint -d /dev/mst/mt4117_pciconf0 query full > flint_query.txt;
sudo flint -d /dev/mst/mt4117_pciconf0 hw query > flint_hwinfo.txt;
sudo flint -d /dev/mst/mt4117_pciconf0 ri orig_firmware.bin;
sudo flint -d /dev/mst/mt4117_pciconf0 dc orig_firmware.ini;
sudo mlxburn -d /dev/mst/mt4117_pciconf0 -vpd > orig_vpd.txt;
Make sure to backup these files in a safe place, especially if you're using a live OS like I am.

Step 3: Download the Latest Firmware
Download the latest firmware for the MCX4121A-ACAT, and unzip it :
Bash:
wget https://www.mellanox.com/downloads/firmware/fw-ConnectX4Lx-rel-14_32_1900-MCX4121A-ACA_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin.zip
unzip fw-ConnectX4Lx-rel-14_32_1900-MCX4121A-ACA_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin.zip
Step 4: Flash the Firmware
First we disable write protection :
sudo flint -d /dev/mst/mt4117_pciconf0 -override_cache_replacement hw set Flash0.WriteProtected=Disabled

Then, choose one of the two flashing options :

1. If you just want to update to the Mellanox firmware, not concerned about some data still showing the HPE origin of the card :
sudo flint -allow_psid_change -d /dev/mst/mt4117_pciconf0 -i fw-ConnectX4Lx-rel-14_32_1900-MCX4121A-ACA_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin burn

2. If you want to erase any trace of it being an HPE card :
sudo flint -nofs --use_image_ps --ignore_dev_data -allow_psid_change -d /dev/mst/mt4117_pciconf0 -i fw-ConnectX4Lx-rel-14_30_1004-MCX4121A-ACA_Ax-UEFI-14.23.17-FlexBoot-3.6.301.bin burn
Beware in that case, Serial Number, GUID and MAC Address will be erased.

The difference between the two seems to be purely cosmetic, as far as I know.

Step 5: Load the New Firmware
After flashing, you will see an information line with instructions on how to load the new firmware. For example :
-I- To load new FW run mlxfwreset or reboot machine.
In the above case, a simple sudo mlxfwreset -d /dev/mst/mt4117_pciconf0 reset will do, no reboot needed.
Sometimes, only a reboot will be offered (when doing full erase for instance).

After power cycle, if you used the second command (as I did initially, blindly following a procedure), you can restore GUID and MAC.

Step 6: Restore GUID and MAC (Optional)
Many posts and docs reference that command, with the -ocr parameter to restore the GUID and MAC :
sudo flint -d /dev/mst/mt4117_pciconfx 0x<your guid> -mac 0x<your mac> sg
then
sudo mlxfwreset -d /dev/mst/mt4117_pciconf0 reset to reset the card.
The command itself works with no error, but it does nothing — the GUID and MAC are still shown as N/A.

In this thread the command is used without the -ocr parameter, and it works but I get :
Code:
Description:           UID                GuidsNumber
Base GUID:             xxxxxxxxxxxxxxxx        4
Orig Base GUID:        N/A                     4
Base MAC:              yyyyyyyyyyyy            4
Orig Base MAC:         N/A
The thing is, with HPE firmware, there was no Orig Base GUID or Orig Base MAC.

Then I found this post with the same command, but with the smg command instead of the sg one, with comment stating "Set device manufacture GUIDs and MACs".
This seems to be the secret sauce :
sudo flint -d <device> -ocr -guid 0x<your guid> -mac 0x<your mac> smg, then the reset command, and I get something identical to the original HPE :
Code:
Description:           UID                GuidsNumber
Base GUID:             xxxxxxxxxxxxxxxx        4
Base MAC:              yyyyyyyyyyyy            4
No more Orig Base GUID and Orig Base MAC.

Step 7: Full Script
Finally, I wanted to script most of it, so based on this post from reddit and tweaking it a bit, I got this :

Bash:
#!/bin/bash

# Paste latest MFT download URL here :
mft_url='https://www.mellanox.com/downloads/MFT/mft-4.31.0-149-x86_64-deb.tgz'

# Paste latest firmware URL here :
fw_url='https://www.mellanox.com/downloads/firmware/fw-ConnectX4Lx-rel-14_32_1900-MCX4121A-ACA_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin.zip'

--- 

sudo apt update
sudo apt -y install gcc make dkms unzip wget

mft=$(basename "${mft_url}" .tgz)
fw=$(basename "${fw_url}" .zip)

# Download Mellanox Firmware Tools (MFT)
wget $mft_url
if [ ! -f "$mft.tgz" ]; then
    echo "MFT file download error"
    exit 1
fi

tar xvf $mft.tgz
sudo ./$mft/install.sh --oem

if ! command -v mst &> /dev/null; then
    echo "mst command not found"
    exit 1
fi


# Download Firmware
wget $fw_url
if [ ! -f "$fw.zip" ]; then
    echo "Firmware file download error"
    exit 1
fi

unzip $fw.zip

# Start MST service
sudo mst start
sudo mst status

# Get device path & serial :
device=$(sudo mst status | awk '/\/dev\/mst\// {print $1}')
serial=$(sudo mlxburn -d $device -vpd | awk '$1 == "SN" {print $NF}')
bck_folder="./mlx_bck/Serial_${serial}"

# Backup
mkdir -p $bck_folder
sudo flint -d $device query full | tee $bck_folder/flint_query.txt
sudo flint -d $device hw query | tee $bck_folder/flint_hwinfo.txt
echo 'Please wait, backing up original firmware.'
sudo flint -d $device ri $bck_folder/orig_firmware.bin
sudo flint -d $device dc $bck_folder/orig_firmware.ini
sudo mlxburn -d $device -vpd | tee $bck_folder/orig_vpd.txt

# Get MAC && GUID
sudo flint -d $device query | grep MAC
sudo flint -d $device query | grep GUID

# Disable write protection
sudo flint -d $device -ocr hw set Flash0.WriteProtected=Disabled

# Flash firmware
sudo flint -allow_psid_change -d $device -i $fw burn
Really simple, nothing world-changing here, but proceed with caution though.
All you need is retrieve and paste the latest MFT tools and latest firmware download URLs in the quotes of the first 2 variables, and you’re good to go.
All backup files are stored in the ./mlx_bck/Serial_<serial_number> folder, so make sure to copy them to a safe location.
You’ll need to confirm the flash at the end, as I didn’t want to force it without confirmation.
Of course this is with flash option 1, so nothing about GUID and MAC — which require manual steps anyway.
After the flash, a simple sudo mlxfwreset -d $device reset should do if the mlxfwreset option is offered. Otherwise make sure to reboot.



Hope it helps.
Regards.
 
Last edited:

richardm

Member
Sep 27, 2013
47
16
8
Does cross-flashing unlock any capabilities? I have a DELL variant and I've yet to determine what's to be gained by doing this...

Device Type: ConnectX4LX
Part Number: 020NJD_0MRT0D_Ax
Description: Mellanox 25GBE 2P ConnectX-4 Lx Adapter
PSID: DEL2420110034
PCI Device Name: 06:00.0
Base MAC: 0c42a1462E06
Versions: Current Available
FW 14.32.2004 N/A
PXE 3.6.0502 N/A
UEFI 14.25.0018 N/A