ConnectX-3 custom firmware files

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

john389

Member
May 21, 2022
36
12
8
Hi,

I have a few ConnectX-3 cards of type MCX311A and managed, after a lot of research, to find a way to enable SR-IOV support using the custom firmware files (.mlx), which allowed me to include a modified config and thereby create a new bin image to burn to the network adapter.

Sadly the last version of the custom firmware I could find was

ConnectX3-rel-2_40_5030.tgz

Does anyone have a newer version they could share?


For anyone else who is on the same journey, here are my notes.

Important: Use these instructions at your own risk! If your pc crashes while flashing or you flash the wrong firmware to the wrong adapter and it is bricked afterwards, ...

A quick overview:
  1. Modification of the running firmware by usage of `mlxconfig` or `mstconfig` does not work for the model MCX311A-XCAT.
  2. None of the bin firmware images I could find had SR-IOV preenabled.
  3. You have to get the .mlx firmware file, modify the existing config, create a new image out of said .mlx file and the config .ini, which you then burn to the adapter.
Preparations:

You need the Mellanox/Nvidia MFT-Tools, as they contain the necessary programs to modify firmware files and burn them to the adapter.

However, not all versions support the old adapter/firmware versions.

The following versions support modification using `.mlx` firmware images using the `mlxburn -fw ...` command:

* mft-4.20.1-14
* mft-4.18.0-106
* mft-4.17.0-106
* mft-4.10.0-104

However, `mft-4.10.0-104` was the only one to be working for firmware version `ConnectX3-rel-2_40_5030`, otherwise you'll get the error message:

Code:
-E- Image generation is not supported (can not find mic tool). This may be due to an MFT installation using --without-image-generation option or a corrupted installation.
Here are links to this MFT version (better keep a copy archived should they also vanish):

http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-rpm.tgz
http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-deb.tgz

And here is the firmware archive:

http://content.mellanox.com/firmware/ConnectX3-rel-2_40_5030.tgz

Download the MFT-Tools and install them, then run

Code:
mst start

Backup existing firmware:

Use `lspci` to find the PCI bus path of your network adapter, i.e. the first column. For me it was 03:00.0 . Replace it in the script below before running it!

Code:
lspci

# Adjust !!!
id="03:00.0"

mkdir /tmp/mlx-backup
cd /tmp/mlx-backup
# basic info such as GUIDs, MACs, etc.:
if ! flint -d "$id" query full > flint_query.txt
then
    echo -- flint -d "$id" query full failed!
fi
# low-level flash chip info:
if ! flint -d "$id" hw query > flint_hwinfo.txt
then
    echo -- flint -d "$id" hw query failed!
fi
# FW:
if ! flint -d "$id" ri orig_firmware.bin
then
    echo -- flint -d "$id" ri failed!
fi
# FW configuration:
if ! flint -d "$id" dc orig_firmware.ini
then
    echo -- flint -d "$id" dc failed!
fi
# PXE ROM image (if any):
if ! flint -d "$id" rrom orig_rom.mlx
then
    echo -- flint -d "$id" rrom failed!
fi
# PCI VPD (vital product data):
if ! mlxburn -d "$id" -vpd > orig_vpd.txt
then
    echo -- mlxburn -d "$id" vpd failed!
fi
Modify the config file MCX311A-XCA_Ax.ini from the firmware archive by appending the following to the [HCA] section:

Code:
num_pfs = 1
total_vfs = 64
sriov_en = true
Create the bin image for burning:

Code:
mlxburn -fw fw-ConnectX3-rel.mlx -conf MCX311A-XCA_Ax.ini -wrimage fw-ConnectX3-SR-IOV.bin
This .bin firmware file can be written to the adapter using newer versions of MFT-Tools and mlxup.

Verify PSID (that your firmware matches the card you want to write it to):

Code:
lspci

# Adjust !!!
id="03:00.0"
fwimg=/path/to/fw-ConnectX3-SR-IOV.bin

# Get PSID from currently running firmware image of the card
flint -d "$id" query full  | grep PSID

# Get PSID from new firmware image
flint -i "$fwimg" query full | grep PSID

flint -i "$fwimg" query full
flint -i "$fwimg" verify
Write new image to device

Code:
lspci

# Adjust !!!
id="03:00.0"
fwimg=/path/to/fw-ConnectX3-SR-IOV.bin

flint -d "$id" -i "$fwimg" burn
After a reboot, you need to set kernel module parameters, so that SR-IOV is enabled:

Code:
cat <<'EHLO' > /etc/modprobe.d/mlx4_core.conf
# port_type_array: comma separated list of port types, i.e. ethernet=2
# num_vfs: comma separated list of vfs for each port
# probe_vf: comma separated list of vfs that are to be bound/activated on the hypervisor
options mlx4_core port_type_array=2 num_vfs=16 probe_vf=2
EHLO
Reload driver, if already loaded:

Code:
modprobe -r mlx4_{ib,en,core}
modprobe mlx4_core

For anyone who wants to remove the PXE rom:

Code:
lspci

# Adjust !!!
id="03:00.0"

flint -d "$id" --allow_rom_change drom
 
  • Like
Reactions: istamov

john389

Member
May 21, 2022
36
12
8
Hi, thank you for linking those threads. I haven't looked at most of them yet, but my problem with your fs2_update_ini.py was, that while it seemingly succeeded in modifying the bin image for my MCX311A, it did not lead to SR-IOV activation.

Now there are three options:
  1. I did something wrong when using the script
  2. fs2_update_ini.py didn't work correctly - the md5sums of the bin files, unmodified and modified version, are different
  3. newer firmware versions have a block in them that prevents SR-IOV from being activated for my card.
I honestly don't know which it is.

I'm aware that there are lots of threads that deal with this topic in some way or another. But I haven't been able to find newer custom firmware download links, which is why I asked here.

Regarding my notes, I thought they would help others, because I had to assemble that information from lots of different sources, which took time.
 

i386

Well-Known Member
Mar 18, 2016
4,220
1,540
113
34
Germany
Did you test sriov that way?
I remember that a lot of users had problems with it in esxi.
 

BeTeP

Well-Known Member
Mar 23, 2019
653
429
63
The ini update script does what it is supposed to do - it replaces the ini section of the firmware. Nothing more.
The 2.42.5032 release absolutely supports SRIOV. The only reason (besides you making some silly mistake with the ini edit) I can think of is that SRIOV is disabled in the config (which is different from INI. Think of INI as default values and config as active values. Config takes precedence).

you can check current config with mstconfig query command. I usually completely reset the config after flashing. Specially when jumping over so many versions.
 
  • Like
Reactions: CyrilB

john389

Member
May 21, 2022
36
12
8
i386: It's working for me under linux with kernel 5.15.88, KVM, qemu, libvirt. As I don't use ESXI, I can't tell you if there are issues with it. Windows 10 virtual machines don't work with ConnectX-3 SR-IOVs - at least in the case of this adapter. It's a known issue for some time. Linux guests are fine, FreeBSD I think are too, but I'd have to check again.

BeTeP: If you read through what I wrote once more you'll see, that I never said, that the firmware itself is not capable of SR-IOV - I am quite sure it is. I honestly don't know why you had to mention a "silly" mistake. I already stated, that something didn't work and I did not exclude a fault on my part. However, I have spent a lot of time with this problem already and tried it multiple times, so the risk of me having made a mistake are somewhat lowered. As I already wrote at the beginning of my first post, the mstconfig command does _not_ work for my card. From what I researched there seems to be a resource limitation on the hardware side, i.e. the adapter. It would have been wonderful, if I could have just used it to turn SR-IOV on.
 

BeTeP

Well-Known Member
Mar 23, 2019
653
429
63
The config feature is relatively new. Have you tried running the mstconfig AFTER flashing?

As for using the "silly mistake", that's my way of saying "hey buddy, have you double-checked your ini edit? maybe you just made a typo?" as opposed to "hey dumb ****, you clearly do not know what you are doing".
 

john389

Member
May 21, 2022
36
12
8
Regarding mstconfig:

First with the working SR-IOV firmware (MLX):

Code:
root@debian:~/mft-4.23.0-104-x86_64-deb# flint -d 05:00.0 q
Image type:            FS2
FW Version: 2.40.5030
FW Release Date: 4.1.2017
Device ID: 4099
Description: Node Port1 Port2 Sys image
GUIDs: ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
MACs: . . .
VSD:
PSID: MT_117011002

root@debian:~/dmesg | grep mlx
. . .
[  785.745986] mlx4_core: Mellanox ConnectX core driver v4.0-0
[  785.746021] mlx4_core: Initializing 0000:05:00.0
[ 792.660432] mlx4_core 0000:05:00.0: DMFS high rate steer mode is: disabled performance optimized steering
[ 792.660708] mlx4_core 0000:05:00.0: Enabling SR-IOV with 16 VFs
[ 792.770962] mlx4_core: Initializing 0000:05:00.1
[ 792.771528] mlx4_core 0000:05:00.1: enabling device (0000 -> 0002)
[ 792.772624] mlx4_core 0000:05:00.1: Detected virtual function - running in slave mode
[ 792.772651] mlx4_core 0000:05:00.1: PF is not ready - Deferring probe
. . .

root@debian:~/mft-4.23.0-104-x86_64-deb# mstconfig -d 05:00.0 query

Device #1:
----------

Device type: ConnectX3
Device: 05:00.0

Configurations: Next Boot
-E- Failed to query device current configuration
Now modifying (your ini script) and installing the newest available firmware http://www.mellanox.com/downloads/f..._5000-MCX311A-XCA_Ax-FlexBoot-3.4.752.bin.zip for the adapter MCX311A-XCAT (release date: 7-Sep-17).

Code:
# Diff of the custom firmware ini and the extracted ini of the newest firmware bin image:

root@debian:/tmp# diff -u ConnectX3-rel-2_40_5030/MCX311A-XCA_Ax.ini orig_firmware.ini
--- ConnectX3-rel-2_40_5030/MCX311A-XCA_Ax.ini  2017-01-04 16:37:14.000000000 +0000
+++ orig_firmware.ini 2023-03-17 10:35:10.576795345 +0000
@@ -1,4 +1,4 @@
-;; Generated automatically by iniprep tool on Wed Jan 04 18:28:34 IST 2017 from ./cx3-1_MCX311A.prs
+;; Generated automatically by iniprep tool on Tue Sep 05 14:05:13 IDT 2017 from ./cx3-1_MCX311A.prs
;;
;; $Id$

# My ini modifications:

root@debian:/tmp# diff -u orig_firmware.ini{.bak,}
--- orig_firmware.ini.bak       2023-03-17 10:35:18.264795676 +0000
+++ orig_firmware.ini 2023-03-17 10:40:38.768809468 +0000
@@ -40,6 +40,9 @@
i2c2_slave_en=1
led_gt_mode_en=1
blinking_log_led_mode=1
+num_pfs = 1
+total_vfs = 64
+sriov_en = true

[IB]
mlpn_en_port0 = true


root@debian:/tmp# ./mft-scripts/fs2_update_ini.py fw-ConnectX3-rel-2_42_5000-MCX311A-XCA_Ax-FlexBoot-3.4.752.bin orig_firmware.ini
root@debian:/tmp#

root@debian:/tmp# md5sum fw-ConnectX3-rel-2_42_5000-MCX311A-XCA_Ax-FlexBoot-3.4.752.bin*
0d1d3ca23cbc6d1d808bbafc6949a58c  fw-ConnectX3-rel-2_42_5000-MCX311A-XCA_Ax-FlexBoot-3.4.752.bin
d2584f656a725b6fdd5538250c7f7bf2 fw-ConnectX3-rel-2_42_5000-MCX311A-XCA_Ax-FlexBoot-3.4.752.bin.bak

root@debian:/tmp# diff -u <(mstflint -i fw-ConnectX3-rel-2_42_5000-MCX311A-XCA_Ax-FlexBoot-3.4.752.bin dc) <(mstflint -i fw-ConnectX3-rel-2_42_5000-MCX311A-XCA_Ax-FlexBoot-3.4.752.bin.bak dc)
--- /dev/fd/63  2023-03-17 10:43:42.176817360 +0000
+++ /dev/fd/62 2023-03-17 10:43:42.176817360 +0000
@@ -40,9 +40,6 @@
i2c2_slave_en=1
led_gt_mode_en=1
blinking_log_led_mode=1
-num_pfs = 1
-total_vfs = 64
-sriov_en = true

[IB]
mlpn_en_port0 = true

# The ini script did succeed in replacing the ini, at least from what I can see.

root@debian:/tmp# mstflint -d 05:00.0 -i fw-ConnectX3-rel-2_42_5000-MCX311A-XCA_Ax-FlexBoot-3.4.752.bin burn

Current FW version on flash: 2.40.5030
New FW version: 2.42.5000

Burn process will not be failsafe. No checks will be performed.
ALL flash, including the Invariant Sector will be overwritten.
If this process fails, computer may remain in an inoperable state.

Do you want to continue ? (y/n) [n] : y
Burning FS2 FW image without signatures - OK
Restoring signature - OK

# Reboot

root@debian:~# mstflint -d 05:00.0 q
Image type:            FS2
FW Version: 2.42.5000
FW Release Date: 5.9.2017
Product Version: 02.42.50.00
Rom Info: type=PXE version=3.4.752
Device ID: 4099
Description: Node Port1 Port2 Sys image
GUIDs: ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
MACs: . . .
VSD:
PSID: MT_1170110023

# at 120 I added the modprobe information for enabling SR-IOV and reloaded the drivers:

root@debian:~# dmesg | grep mlx4
[    1.999062] mlx4_core: Mellanox ConnectX core driver v4.0-0
[    1.999095] mlx4_core: Initializing 0000:05:00.0
[ 9.022796] mlx4_core 0000:05:00.0: DMFS high rate steer mode is: disabled performance optimized steering
[ 9.023153] mlx4_core 0000:05:00.0: 31.504 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x4 link)
[ 9.257947] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0
[ 9.258105] mlx4_en 0000:05:00.0: Activating port:1
[ 9.263899] mlx4_en: 0000:05:00.0: Port 1: Using 32 TX rings
[ 9.263901] mlx4_en: 0000:05:00.0: Port 1: Using 16 RX rings
[ 9.264140] mlx4_en: 0000:05:00.0: Port 1: Initializing port
[ 9.264557] mlx4_en 0000:05:00.0: registered PHC clock
[ 9.265058] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v4.0-0
[ 9.266003] mlx4_core 0000:05:00.0 enp5s0: renamed from eth0
[ 9.267585] <mlx4_ib> mlx4_ib_add: counter index 1 for port 1 allocated 1
[ 21.455418] mlx4_en: enp5s0: Steering Mode 1
[ 112.549920] mlx4_en: enp5s0: Close port called
[ 112.642656] mlx4_en 0000:05:00.0: removed PHC
[ 113.852395] mlx4_core: Mellanox ConnectX core driver v4.0-0
[ 113.852419] mlx4_core: Initializing 0000:05:00.0
[ 120.834392] mlx4_core 0000:05:00.0: DMFS high rate steer mode is: disabled performance optimized steering

[ 120.834656] mlx4_core 0000:05:00.0: requested vfs (16) > available vfs (0). Continuing without SR_IOV  <---  !!!

[ 120.835724] mlx4_core 0000:05:00.0: 31.504 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x4 link)
[ 121.084019] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0
[ 121.084187] mlx4_en 0000:05:00.0: Activating port:1
[ 121.089751] mlx4_en: 0000:05:00.0: Port 1: Using 32 TX rings
[ 121.089752] mlx4_en: 0000:05:00.0: Port 1: Using 16 RX rings
[ 121.089988] mlx4_en: 0000:05:00.0: Port 1: Initializing port
[ 121.090416] mlx4_en 0000:05:00.0: registered PHC clock
[ 121.093382] mlx4_core 0000:05:00.0 enp5s0: renamed from eth0
[ 121.099762] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v4.0-0
[ 121.102511] <mlx4_ib> mlx4_ib_add: counter index 1 for port 1 allocated 1
[ 121.232822] mlx4_en: enp5s0: Steering Mode 1

root@debian:~# mstconfig -d 05:00.0 q

Device #1:
----------

Device type: ConnectX3
Device: 05:00.0

Configurations: Next Boot
-E- Failed to query device current configuration

root@debian:~# mstconfig -d 05:00.0 s SRIOV_EN=True

Device #1:
----------

Device type: ConnectX3
Device: 05:00.0

Configurations: Next Boot New
-E- Failed to query device current configuration

root@debian:~# mstflint -d 05:00.0 dc | grep -A16 HCA
[HCA]
hca_header_device_id = 0x1003
hca_header_subsystem_id = 0x0055
hca_header_class_code = 0x020000
dpdp_en = false
eth_xfi_en = true
mdio_en_port1 = 0
pcie_tx_polarity = 0x0f
gpio_swap_enable=1
module_i2c_bus_select_port1=SECONDARY_BUS
i2c2_slave_en=1
led_gt_mode_en=1
blinking_log_led_mode=1
num_pfs = 1
total_vfs = 64
sriov_en = true
I honestly don't know what I could have done wrong.

The facts remain, even after spending another hour with this stuff:

  1. mstconfig still doesn't work with my card, no matter the firmware version
  2. The ini modification using your script seems to work but does not lead to activated SR-IOV.
  3. Only the old MLX firmware version, which I modified in the same way as the newer bin firmware, with the exact same added config lines, allows me to enable SR-IOV on this older card.
 

BeTeP

Well-Known Member
Mar 23, 2019
653
429
63
My gut feeling still tells me that malfunctioning config feature might be related to not being able to properly enable other features like SRIOV.
So I would either try to fix the config or propely disable it (it's enabled in the 2.42.5000).

Code:
[ADAPTER]
nv_cfg_en = false
nv_cfg_memfree_write_en = false
nv_cfg_memfree_use_scratchpad = false
nv_cfg_default_param_en = false
 

john389

Member
May 21, 2022
36
12
8
You understand, that the config feature never works, no matter if I change the config or not, independent of firmware version?

That means that the default config with which the new firmware comes also exhibits the problem, meaning unless the default config, which didn't change between 2.40.5030 and 2.42.5000 !!, is somehow wrong for the adapter, then there is really nothing I can do to get the config feature working.

If you look at these links, then it seems I am not the only one with this issue and everything points to the device id 4099 as the source of the problem.

mstconfig not working w/ ConnectX-3 · Issue #590 · Mellanox/mstflint
mstconfig Fails to query device configuration on Connectx-3 EN cards · Issue #33 · Mellanox/mstflint
Unable to set Mellanox ConnectX-3 to Ethernet (Failed to query device current configuration)

Although I can't find it at the moment, I recall there was some discussion in some forum, that pointed to missing resources - think RAM/flash space - on the one-port ConnectX-3 card, that led to the config feature being inoperable.

Considering that editing the config for the older version works, do you think that the cfg feature somehow interferes with the config changes only for the newer firmware version - the older one worked fine?

I don't currently have one of these cards available for further testing. If someone else wants to try, I would be grateful if you'd share the results.
 
  • Like
Reactions: 2bluesc

TheRac25

New Member
May 30, 2023
12
0
1
2.42.5000 SRIOV is still "broken" with nv_cfg disabled
Code:
[    8.137305] mlx4_core 0000:01:00.0: requested vfs (16) > available vfs (0). Continuing without SR_IOV
this seems to be how they encourage people to buy new hardware, perfect fit for nvidia
 

john389

Member
May 21, 2022
36
12
8
Well yes, I tested it with both enabled and disabled and never could get SR-IOV with firmware versions newer than 2.40.5030 to work. So I started buying ConnectX-4 and Intel X710/E810 network cards instead. Sadly they are quite a bit more expensive ...
 

BackupProphet

Well-Known Member
Jul 2, 2014
1,083
640
113
Stavanger, Norway
olavgg.com
I am struggling with the same problem now with a Dell CX322A , which is a Mellanox ConnectX3 Pro and I using the MCX312C-XCC_Ax ini file. I cannot get SR-IOV to work at all, and even RDMA/ROCE seems to be non functional.

If I downflash to 2_40_5030 the sr-iov option disappear
Code:
mstconfig -d 03:00.0 q

Device #1:
----------

Device type:    ConnectX3Pro
Device:         03:00.0

Configurations:                              Next Boot
         BOOT_OPTION_ROM_EN_P1               True(1)
         BOOT_VLAN_EN_P1                     False(0)
         BOOT_RETRY_CNT_P1                   0
         LEGACY_BOOT_PROTOCOL_P1             PXE(1)
         BOOT_VLAN_P1                        1
         BOOT_OPTION_ROM_EN_P2               True(1)
         BOOT_VLAN_EN_P2                     False(0)
         BOOT_RETRY_CNT_P2                   0
         LEGACY_BOOT_PROTOCOL_P2             PXE(1)
         BOOT_VLAN_P2                        1
         IP_VER_P1                           IPv4(0)
         IP_VER_P2                           IPv4(0)
I have a feeling that its impossible to restore this to a normal MCX312C
 

BackupProphet

Well-Known Member
Jul 2, 2014
1,083
640
113
Stavanger, Norway
olavgg.com
yes,
Code:
mstflint -d 03:00.0 query
Image type:            FS2
FW Version:            2.40.5048
FW Release Date:       5.3.2017
Product Version:       02.40.50.48
Rom Info:              type=UEFI version=15.11.40 cpu=AMD64
                       type=PXE version=3.4.746
Device ID:             4103
Description:           Node             Port1            Port2            Sys image
GUIDs:                 452140300ab332ba 452140300ab332bb 452140300ab332bc 452140300ab332bd
MACs:                                       ec0d9a0e5360     ec0d9a0e5361
VSD:
PSID:                  DEL2310000023


lspci | grep Mellanox
03:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
 

BackupProphet

Well-Known Member
Jul 2, 2014
1,083
640
113
Stavanger, Norway
olavgg.com
Ok, seems like I've figured out a few things.
According to this post HOW TO: Using SR-IOV in UnRAID with 1Gb/10Gb/40Gb network interface cards (NICs)

Mellanox ConnectX 2 and 3 cannot be used together for RDMA, and if you enable SR-IOV you disable RDMA. So that is why I'm having issues with RDMA functionality. Also there are issues with Windows guests. https://forums.servethehome.com/ind...-and-windows-guest-via-kvm.28956/#post-306213

Seems like if you want a good sysadmin life, just pay extra for ConnectX 4+ or get Intel.
 

ano

Well-Known Member
Nov 7, 2022
634
259
63
can I ask what you use rdma for? and if you need the extra perf?

we are using older cx3 for misc ceph stuff, and iscsi/zfs still, and some smb
 

kevindd992002

Member
Oct 4, 2021
110
4
18
I'll attach my firmware images for anyone who wants to try them.

These can be written to the adapter by using flint from MFT-Tools (newest version works just fine), mstflint or mlxup.
@john389 so I tried the "sriov working" 2.40.5030 fw that you uploaded here but for some reason sriov is still not working? I don't see the vf's under lspci.

EDIT: Ok, so I generated my own 2.40.5030 image and that seems to enable SRIOV working just fine. It looks like the file you uploaded in that post isn't working after all.
 
Last edited: