Hi,
I have a few ConnectX-3 cards of type MCX311A and managed, after a lot of research, to find a way to enable SR-IOV support using the custom firmware files (.mlx), which allowed me to include a modified config and thereby create a new bin image to burn to the network adapter.
Sadly the last version of the custom firmware I could find was
ConnectX3-rel-2_40_5030.tgz
Does anyone have a newer version they could share?
For anyone else who is on the same journey, here are my notes.
Important: Use these instructions at your own risk! If your pc crashes while flashing or you flash the wrong firmware to the wrong adapter and it is bricked afterwards, ...
A quick overview:
You need the Mellanox/Nvidia MFT-Tools, as they contain the necessary programs to modify firmware files and burn them to the adapter.
However, not all versions support the old adapter/firmware versions.
The following versions support modification using `.mlx` firmware images using the `mlxburn -fw ...` command:
* mft-4.20.1-14
* mft-4.18.0-106
* mft-4.17.0-106
* mft-4.10.0-104
However, `mft-4.10.0-104` was the only one to be working for firmware version `ConnectX3-rel-2_40_5030`, otherwise you'll get the error message:
Here are links to this MFT version (better keep a copy archived should they also vanish):
http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-rpm.tgz
http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-deb.tgz
And here is the firmware archive:
http://content.mellanox.com/firmware/ConnectX3-rel-2_40_5030.tgz
Download the MFT-Tools and install them, then run
Backup existing firmware:
Use `lspci` to find the PCI bus path of your network adapter, i.e. the first column. For me it was 03:00.0 . Replace it in the script below before running it!
Modify the config file MCX311A-XCA_Ax.ini from the firmware archive by appending the following to the [HCA] section:
Create the bin image for burning:
This .bin firmware file can be written to the adapter using newer versions of MFT-Tools and mlxup.
Verify PSID (that your firmware matches the card you want to write it to):
Write new image to device
After a reboot, you need to set kernel module parameters, so that SR-IOV is enabled:
Reload driver, if already loaded:
For anyone who wants to remove the PXE rom:
I have a few ConnectX-3 cards of type MCX311A and managed, after a lot of research, to find a way to enable SR-IOV support using the custom firmware files (.mlx), which allowed me to include a modified config and thereby create a new bin image to burn to the network adapter.
Sadly the last version of the custom firmware I could find was
ConnectX3-rel-2_40_5030.tgz
Does anyone have a newer version they could share?
For anyone else who is on the same journey, here are my notes.
Important: Use these instructions at your own risk! If your pc crashes while flashing or you flash the wrong firmware to the wrong adapter and it is bricked afterwards, ...
A quick overview:
- Modification of the running firmware by usage of `mlxconfig` or `mstconfig` does not work for the model MCX311A-XCAT.
- None of the bin firmware images I could find had SR-IOV preenabled.
- You have to get the .mlx firmware file, modify the existing config, create a new image out of said .mlx file and the config .ini, which you then burn to the adapter.
You need the Mellanox/Nvidia MFT-Tools, as they contain the necessary programs to modify firmware files and burn them to the adapter.
However, not all versions support the old adapter/firmware versions.
The following versions support modification using `.mlx` firmware images using the `mlxburn -fw ...` command:
* mft-4.20.1-14
* mft-4.18.0-106
* mft-4.17.0-106
* mft-4.10.0-104
However, `mft-4.10.0-104` was the only one to be working for firmware version `ConnectX3-rel-2_40_5030`, otherwise you'll get the error message:
Code:
-E- Image generation is not supported (can not find mic tool). This may be due to an MFT installation using --without-image-generation option or a corrupted installation.
http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-rpm.tgz
http://www.mellanox.com/downloads/MFT/mft-4.10.0-104-x86_64-deb.tgz
And here is the firmware archive:
http://content.mellanox.com/firmware/ConnectX3-rel-2_40_5030.tgz
Download the MFT-Tools and install them, then run
Code:
mst start
Backup existing firmware:
Use `lspci` to find the PCI bus path of your network adapter, i.e. the first column. For me it was 03:00.0 . Replace it in the script below before running it!
Code:
lspci
# Adjust !!!
id="03:00.0"
mkdir /tmp/mlx-backup
cd /tmp/mlx-backup
# basic info such as GUIDs, MACs, etc.:
if ! flint -d "$id" query full > flint_query.txt
then
echo -- flint -d "$id" query full failed!
fi
# low-level flash chip info:
if ! flint -d "$id" hw query > flint_hwinfo.txt
then
echo -- flint -d "$id" hw query failed!
fi
# FW:
if ! flint -d "$id" ri orig_firmware.bin
then
echo -- flint -d "$id" ri failed!
fi
# FW configuration:
if ! flint -d "$id" dc orig_firmware.ini
then
echo -- flint -d "$id" dc failed!
fi
# PXE ROM image (if any):
if ! flint -d "$id" rrom orig_rom.mlx
then
echo -- flint -d "$id" rrom failed!
fi
# PCI VPD (vital product data):
if ! mlxburn -d "$id" -vpd > orig_vpd.txt
then
echo -- mlxburn -d "$id" vpd failed!
fi
Code:
num_pfs = 1
total_vfs = 64
sriov_en = true
Code:
mlxburn -fw fw-ConnectX3-rel.mlx -conf MCX311A-XCA_Ax.ini -wrimage fw-ConnectX3-SR-IOV.bin
Verify PSID (that your firmware matches the card you want to write it to):
Code:
lspci
# Adjust !!!
id="03:00.0"
fwimg=/path/to/fw-ConnectX3-SR-IOV.bin
# Get PSID from currently running firmware image of the card
flint -d "$id" query full | grep PSID
# Get PSID from new firmware image
flint -i "$fwimg" query full | grep PSID
flint -i "$fwimg" query full
flint -i "$fwimg" verify
Code:
lspci
# Adjust !!!
id="03:00.0"
fwimg=/path/to/fw-ConnectX3-SR-IOV.bin
flint -d "$id" -i "$fwimg" burn
Code:
cat <<'EHLO' > /etc/modprobe.d/mlx4_core.conf
# port_type_array: comma separated list of port types, i.e. ethernet=2
# num_vfs: comma separated list of vfs for each port
# probe_vf: comma separated list of vfs that are to be bound/activated on the hypervisor
options mlx4_core port_type_array=2 num_vfs=16 probe_vf=2
EHLO
Code:
modprobe -r mlx4_{ib,en,core}
modprobe mlx4_core
For anyone who wants to remove the PXE rom:
Code:
lspci
# Adjust !!!
id="03:00.0"
flint -d "$id" --allow_rom_change drom