Enabling PCIe Gen4 on ConnectX-5 Cards — Firmware Patch Tool & Full Writeup
I could only track down a Dell updater that went as far as firmware 16.35.4554, but I wanted to see how the latest LTS build (16.35.8002) would behave. I jumped the FNP pins on both cards to enter firmware recovery mode and flashed the 8002 LTS image for the MCX512F-ACAT (stock Mellanox). Both cards immediately stopped negotiating links correctly, failures to auto-negotiate on both, and no amount of fiddling with mstconfig or mlxlink resolved it. Flashing back to the Dell OEM image resolved all issues and functioned as expected.. I then flashed the HPE card with the Dell image (again from recovery mode) and that also worked without issue. So both cards run happily on Dell OEM firmware.
That was the first interesting data point. The second came from a few forum posts I found where people mentioned crossflashing CX5-EN cards to CX5-Ex firmware to get PCIe Gen4. The closest SFP28 card with Gen4 support is the MCX512A-ADAT (CX5-Ex EN), which has a physical x8 interface. The only Gen4 card I could find with a physical x16 interface was the MCX516A-ADAT, a 100GbE QSFP28 card. That got me wondering: what's actually different in the firmware between Gen3 and Gen4 cards if NVIDIA just segments at the firmware level and charges a premium for the Ex.
Why Stock Mellanox Firmware Sometimes Breaks on OEM Cards
Before jumping into PCIe Gen4, it's worth noting why the stock FW didn't work because it affects anyone working with OEM CX5 cards.
The Dell CX512F uses a Socket Direct PCIe configuration — 2×8 lanes instead of standard x16. This is baked into the firmware along with Dell-specific PHY equalization coefficients, trace compensation values, SFP link tuning parameters, and a higher board power budget (15.9W vs 12.5W for stock). When you flash the stock Mellanox ACAT image, which expects a standard x16 PCIe layout, the mismatch in PCIe port mode and PHY configuration causes the card to fail at link negotiation.
Binary diffing the Dell and stock images revealed 95 bytes of functional configuration differences spread across four firmware sections, PHY EQ coefficients tuned for the Dell PCB layout, Socket Direct port mode settings, SFP signal quality parameters, PHY calibration values specific to Dell's trace routing, and speed table profiles that map to the Dell x8 port arrangement. None of these are identity fields (PSID, name strings), they're board-level electrical configuration that has to match the physical hardware.
How the Firmware Was Reverse-Engineered
I used Claude for the binary analysis, diffing, and code generation. I drove the direction, supplied the firmware images, tested on hardware.
I uploaded multiple firmware images for comparison:
The first pass compared the ACAT (Gen3) and ADAT (Gen4) images at the same firmware version to isolate what changes between EN and Ex. Then compared the Dell OEM image against stock Mellanox to understand what Dell customized. The Dell firmware was particularly useful because Dell's OEM build happened to have some Gen4-adjacent configuration values already set differently from stock Mellanox, which helped narrow down the search space.
Mellanox's open-source mstflint tool can dump named configuration fields with mstflint dc, but it turned out that only 1 of the 8 critical Gen4 bytes shows up as a named field (pcie_cfg.pcie_max_speed_supported). The other 7 are in unnamed regions of the config sections that mstflint treats as opaque binary. Raw hex diffing was the only path to find them.
The 8-Byte Gen4 Patch
After a lot of iterative diffing, patching, flashing, and testing, the minimum viable Gen4 patch came down to exactly 8 bytes across three firmware config sections. Every byte is defined by its offset within a section, not by absolute position in the image, which makes this firmware-version-independent.
Two of these bytes are universal — they control Gen4 on every CX5 variant regardless of connector type or port count:
FW_BOOT_CFG section:
HW_BOOT_CFG section:
The remaining six bytes are SFP28-specific. QSFP28 and single-port cards already have correct values from the factory (see How It Works Across Card Types below).
HW_MAIN_CFG section:
Each of these serves a distinct purpose in the Gen4 enablement chain:
Capability advertisement (+0x0093): This controls what the card reports in its PCIe Express Capability Structure during enumeration. If the card advertises Gen3 max, the root complex won't even attempt Gen4 link training. Changing from 0x45 to 0x47 makes the card advertise Gen4 capability to the host.
Link training mode (+0x0023): This is the one field that mstflint dc actually knows by name. Value 0x07 constrains the PHY to Gen3 link training sequences. Value 0x0F enables Gen4 link training, which includes the Phase 2 and Phase 3 equalization steps that PCIe 4.0 requires for reliable 16 GT/s signaling. This byte was the last one discovered and the hardest to find — everything else could be set correctly and the card would still train at Gen3 without it.
Port generation targets (+0x0245 / +0x0285): These tell the firmware what PCIe generation to configure the PHY for during initialization. Value 0x01 = Gen3, 0x04 = Gen4. Without these, the PHY never targets Gen4 speed. On QSFP28 cards, this value is 0x06 for both Gen3 and Gen4 — the tool skips these automatically.
Speed table invalidation (+0x0404–0x0407): The firmware has an internal lookup table that maps speed profile indices to PCIe configurations. Profiles 0x0020 and 0x0021 are Gen3-only fallback profiles. If they remain valid, the link training state machine can fall back to them even with everything else set to Gen4. Writing 0x0FFF to both entries marks them as invalid, preventing Gen3 fallback. On QSFP28 cards, these ship pre-invalidated at 0x0FFF already.
How It Works Across Card Types
The tool detects your card type automatically and applies only the patches that are needed:
SFP28 cards (MCX512F, MCX512A): All 8 bytes are patched — port generation, capability index, speed tables, and link training mode.
QSFP28 cards (MCX555A, MCX515A, MCX516A): Only 2 bytes need patching — capability index and link training mode. Port generation bytes use 0x06 for both Gen3 and Gen4 on QSFP28, and speed tables ship pre-invalidated. The tool detects this and skips accordingly.
Single-port cards: Port 2 generation byte is 0x00 (disabled). The tool skips it automatically.
What About the Device ID?
Early on, we went down a dead-end investigating whether Gen4 was gated behind the Device ID (0x1017 for CX5-EN vs 0x1019 for CX5-Ex). Some crossflash guides suggest this, and the ADAT firmware does carry 0x1019. Initial experiments seemed to confirm it, but the real problem was that we hadn't found all 8 bytes yet. Once the boot config byte and speed table invalidation were discovered, Gen4 worked without touching the Device ID at all.
This was confirmed across three different firmware/card combinations, all keeping the original 0x1017 Device ID:
CRC Recalculation
Changing any byte in a firmware section invalidates its CRC. The FS4 format stores two CRC values per section: one over the section data itself, and one over the ITOC entry (which includes the section CRC). Both need to be recalculated after patching.
Mellanox uses a non-standard CRC-16 with polynomial 0x100B. This isn't any of the usual suspects (CCITT, XMODEM, Modbus, etc.) I spent time testing against those before going straight to the source. The implementation lives in mft_utils/crc16.cpp in the open-source mstflint repo. Init value 0xFFFF, processes 32-bit big-endian words, finalizes with a 16-bit zero flush and XOR 0xFFFF. The Python tool implements this natively, so there's no runtime dependency on mstflint for patching.
OEM Firmware Upgrade
As I mentioned at the top, flashing stock Mellanox 8002 onto OEM cards broke things because of the 95 bytes of board-specific configuration. What I wanted was the latest LTS firmware engine with the Dell board tuning intact and Gen4 enabled.
The tool supports this directly with the --upgrade-base flag. You feed it your OEM image (for vendor detection) and a stock Mellanox LTS image (as the new base). It automatically applies the appropriate vendor profile, all 95 Dell customization bytes covering PHY EQ, PCIe port mode, power budget, SFP tuning, and calibration values, onto the LTS base, then applies the Gen4 patch on top.
The Dell OEM profile is built into the tool. Other vendor profiles (HPE, Lenovo, Supermicro) can be added as the community provides firmware samples to diff against their stock equivalents.
The Tool
I initially thought about uploading pre-patched firmware images, but that's a terrible idea for network hardware. You'd be trusting a random binary from a random person on the internet to handle traffic on your network. Instead, I had Claude make a tool where you supply your own firmware image, the code is fully reviewable, and the tool makes only the documented changes plus CRC recalculation.
Python 3.8+, Parses the FS4 ITOC at runtime (not hardcoded to any firmware version), verifies values before overwriting, recalculates all CRCs natively.
GitHub: Bulls729/Mellanox-ConnectX-5-PCIe-Gen-4-Enablement
Back Up Your Firmware First
Before doing anything else, back up your current working firmware and note your GUIDs/MACs:
Basic Usage (Adjust DeviceID as needed, use mststatus for correct ID retrieval)
Important Note — Signature Errors and FNP Recovery
If you're flashing modified firmware, or flashing an image from a different vendor onto your card (e.g., Dell firmware onto an HPE card, or vice versa), the card's secure boot will reject the image because the signature no longer matches. The card needs to be in FNP (Firmware Not Present) recovery mode to bypass the signing check.
Common error messages that indicate you need FNP recovery mode:
How to enter FNP recovery mode: Short the FNP jumper pins on the card before powering on. The card will appear as MT28800 Family [ConnectX-5 Flash Recovery] (PCI ID 15b3:020d) in lspci. Flash from this state, remove the jumper, then do a full power cycle (not reboot). For cards without an accessible FNP header, shorting SPI flash pins 2 and 4 during boot achieves the same recovery state.
Current Status & Tested Hardware
Port status LEDs function correctly on both tested cards. I've seen posts where people have had LED issues after firmware modifications — they're working as expected here on both the Dell and HPE hardware.
Community Help Needed
The tool should handle all CX5 variants but needs further testing on:
Changelog
v1.1.0 — 2026-03-02
Acknowledgments
CRC-16 implementation derived from the mstflint open-source project (dual GPL-2.0/BSD license). Firmware analysis and tool development done with significant assistance from Claude. Tool and all documentation are MIT-licensed.
Edit: Accompanying Reddit post: Enabling PCIe Gen4 on ConnectX-5 EN Cards — Firmware Patch Tool
This started with two purchases: A Dell-branded ConnectX-5 (CX512F, Dell P/N: 0TDNNT) and an HPE ConnectX-5 (CX512F, HPE SPS P/N: P12608). Identical PCBs. Both shipped with older firmware from their respective OEMs.
I could only track down a Dell updater that went as far as firmware 16.35.4554, but I wanted to see how the latest LTS build (16.35.8002) would behave. I jumped the FNP pins on both cards to enter firmware recovery mode and flashed the 8002 LTS image for the MCX512F-ACAT (stock Mellanox). Both cards immediately stopped negotiating links correctly, failures to auto-negotiate on both, and no amount of fiddling with mstconfig or mlxlink resolved it. Flashing back to the Dell OEM image resolved all issues and functioned as expected.. I then flashed the HPE card with the Dell image (again from recovery mode) and that also worked without issue. So both cards run happily on Dell OEM firmware.
That was the first interesting data point. The second came from a few forum posts I found where people mentioned crossflashing CX5-EN cards to CX5-Ex firmware to get PCIe Gen4. The closest SFP28 card with Gen4 support is the MCX512A-ADAT (CX5-Ex EN), which has a physical x8 interface. The only Gen4 card I could find with a physical x16 interface was the MCX516A-ADAT, a 100GbE QSFP28 card. That got me wondering: what's actually different in the firmware between Gen3 and Gen4 cards if NVIDIA just segments at the firmware level and charges a premium for the Ex.
Why Stock Mellanox Firmware Sometimes Breaks on OEM Cards
Before jumping into PCIe Gen4, it's worth noting why the stock FW didn't work because it affects anyone working with OEM CX5 cards.
The Dell CX512F uses a Socket Direct PCIe configuration — 2×8 lanes instead of standard x16. This is baked into the firmware along with Dell-specific PHY equalization coefficients, trace compensation values, SFP link tuning parameters, and a higher board power budget (15.9W vs 12.5W for stock). When you flash the stock Mellanox ACAT image, which expects a standard x16 PCIe layout, the mismatch in PCIe port mode and PHY configuration causes the card to fail at link negotiation.
Binary diffing the Dell and stock images revealed 95 bytes of functional configuration differences spread across four firmware sections, PHY EQ coefficients tuned for the Dell PCB layout, Socket Direct port mode settings, SFP signal quality parameters, PHY calibration values specific to Dell's trace routing, and speed table profiles that map to the Dell x8 port arrangement. None of these are identity fields (PSID, name strings), they're board-level electrical configuration that has to match the physical hardware.
How the Firmware Was Reverse-Engineered
I used Claude for the binary analysis, diffing, and code generation. I drove the direction, supplied the firmware images, tested on hardware.
I uploaded multiple firmware images for comparison:
- Dell OEM 16.35.4554 (MCX512F-ACA, the working Gen3 image)
- Stock Mellanox ACAT 16.35.4554 (MCX512F-ACA)
- Stock Mellanox ACAT 16.35.8002 (MCX512F-ACA)
- Stock Mellanox ADAT 16.35.4554 (MCX512A-ADA — the Gen4 Ex card)
- Stock Mellanox ADAT 16.35.8002 (MCX512A-ADA)
- Stock Mellanox 16.35.8002 (MCX516A-CDA — 100G QSFP28 Gen4 x16)
- Stock Mellanox 16.35.1012 (MCX555A-ECA — VPI single-port QSFP28 Gen3)
The first pass compared the ACAT (Gen3) and ADAT (Gen4) images at the same firmware version to isolate what changes between EN and Ex. Then compared the Dell OEM image against stock Mellanox to understand what Dell customized. The Dell firmware was particularly useful because Dell's OEM build happened to have some Gen4-adjacent configuration values already set differently from stock Mellanox, which helped narrow down the search space.
Mellanox's open-source mstflint tool can dump named configuration fields with mstflint dc, but it turned out that only 1 of the 8 critical Gen4 bytes shows up as a named field (pcie_cfg.pcie_max_speed_supported). The other 7 are in unnamed regions of the config sections that mstflint treats as opaque binary. Raw hex diffing was the only path to find them.
The 8-Byte Gen4 Patch
After a lot of iterative diffing, patching, flashing, and testing, the minimum viable Gen4 patch came down to exactly 8 bytes across three firmware config sections. Every byte is defined by its offset within a section, not by absolute position in the image, which makes this firmware-version-independent.
Two of these bytes are universal — they control Gen4 on every CX5 variant regardless of connector type or port count:
FW_BOOT_CFG section:
| Offset | Stock | Patched | What it does |
|---|---|---|---|
| +0x0093 | 0x45 | 0x47 | PCIe capability advertisement index (advertise Gen4 to host) |
| Offset | Stock | Patched | What it does |
|---|---|---|---|
| +0x0023 | 0x07 | 0x0F | pcie_cfg.pcie_max_speed_supported — enables Gen4 link training with Phase 2/3 EQ |
HW_MAIN_CFG section:
| Offset | Stock | Patched | What it does |
|---|---|---|---|
| +0x0245 | 0x01 | 0x04 | Port 1 PCIe generation target (Gen3 → Gen4) |
| +0x0285 | 0x01 | 0x04 | Port 2 PCIe generation target (Gen3 → Gen4) |
| +0x0404 | 0x00 | 0x0F | Invalidate Gen3 speed profile 0x0020 (high byte) |
| +0x0405 | 0x20 | 0xFF | Invalidate Gen3 speed profile 0x0020 (low byte) |
| +0x0406 | 0x00 | 0x0F | Invalidate Gen3 speed profile 0x0021 (high byte) |
| +0x0407 | 0x21 | 0xFF | Invalidate Gen3 speed profile 0x0021 (low byte) |
Capability advertisement (+0x0093): This controls what the card reports in its PCIe Express Capability Structure during enumeration. If the card advertises Gen3 max, the root complex won't even attempt Gen4 link training. Changing from 0x45 to 0x47 makes the card advertise Gen4 capability to the host.
Link training mode (+0x0023): This is the one field that mstflint dc actually knows by name. Value 0x07 constrains the PHY to Gen3 link training sequences. Value 0x0F enables Gen4 link training, which includes the Phase 2 and Phase 3 equalization steps that PCIe 4.0 requires for reliable 16 GT/s signaling. This byte was the last one discovered and the hardest to find — everything else could be set correctly and the card would still train at Gen3 without it.
Port generation targets (+0x0245 / +0x0285): These tell the firmware what PCIe generation to configure the PHY for during initialization. Value 0x01 = Gen3, 0x04 = Gen4. Without these, the PHY never targets Gen4 speed. On QSFP28 cards, this value is 0x06 for both Gen3 and Gen4 — the tool skips these automatically.
Speed table invalidation (+0x0404–0x0407): The firmware has an internal lookup table that maps speed profile indices to PCIe configurations. Profiles 0x0020 and 0x0021 are Gen3-only fallback profiles. If they remain valid, the link training state machine can fall back to them even with everything else set to Gen4. Writing 0x0FFF to both entries marks them as invalid, preventing Gen3 fallback. On QSFP28 cards, these ship pre-invalidated at 0x0FFF already.
How It Works Across Card Types
The tool detects your card type automatically and applies only the patches that are needed:
SFP28 cards (MCX512F, MCX512A): All 8 bytes are patched — port generation, capability index, speed tables, and link training mode.
QSFP28 cards (MCX555A, MCX515A, MCX516A): Only 2 bytes need patching — capability index and link training mode. Port generation bytes use 0x06 for both Gen3 and Gen4 on QSFP28, and speed tables ship pre-invalidated. The tool detects this and skips accordingly.
Single-port cards: Port 2 generation byte is 0x00 (disabled). The tool skips it automatically.
What About the Device ID?
Early on, we went down a dead-end investigating whether Gen4 was gated behind the Device ID (0x1017 for CX5-EN vs 0x1019 for CX5-Ex). Some crossflash guides suggest this, and the ADAT firmware does carry 0x1019. Initial experiments seemed to confirm it, but the real problem was that we hadn't found all 8 bytes yet. Once the boot config byte and speed table invalidation were discovered, Gen4 worked without touching the Device ID at all.
This was confirmed across three different firmware/card combinations, all keeping the original 0x1017 Device ID:
- Dell OEM firmware on the Dell 0TDNNT card — Gen4 working
- Dell OEM firmware on the HPE P12608 card — Gen4 working
- Stock Mellanox ACAT firmware — Gen4 working
CRC Recalculation
Changing any byte in a firmware section invalidates its CRC. The FS4 format stores two CRC values per section: one over the section data itself, and one over the ITOC entry (which includes the section CRC). Both need to be recalculated after patching.
Mellanox uses a non-standard CRC-16 with polynomial 0x100B. This isn't any of the usual suspects (CCITT, XMODEM, Modbus, etc.) I spent time testing against those before going straight to the source. The implementation lives in mft_utils/crc16.cpp in the open-source mstflint repo. Init value 0xFFFF, processes 32-bit big-endian words, finalizes with a 16-bit zero flush and XOR 0xFFFF. The Python tool implements this natively, so there's no runtime dependency on mstflint for patching.
OEM Firmware Upgrade
As I mentioned at the top, flashing stock Mellanox 8002 onto OEM cards broke things because of the 95 bytes of board-specific configuration. What I wanted was the latest LTS firmware engine with the Dell board tuning intact and Gen4 enabled.
The tool supports this directly with the --upgrade-base flag. You feed it your OEM image (for vendor detection) and a stock Mellanox LTS image (as the new base). It automatically applies the appropriate vendor profile, all 95 Dell customization bytes covering PHY EQ, PCIe port mode, power budget, SFP tuning, and calibration values, onto the LTS base, then applies the Gen4 patch on top.
Code:
python3 cx5_gen4_enable.py \
--input dell_oem_4554.bin \
--upgrade-base fw-ConnectX5-rel-16_35_8002-MCX512F-ACA.bin \
--output dell_tuned_8002_gen4.binThe Tool
I initially thought about uploading pre-patched firmware images, but that's a terrible idea for network hardware. You'd be trusting a random binary from a random person on the internet to handle traffic on your network. Instead, I had Claude make a tool where you supply your own firmware image, the code is fully reviewable, and the tool makes only the documented changes plus CRC recalculation.
Python 3.8+, Parses the FS4 ITOC at runtime (not hardcoded to any firmware version), verifies values before overwriting, recalculates all CRCs natively.
GitHub: Bulls729/Mellanox-ConnectX-5-PCIe-Gen-4-Enablement
Back Up Your Firmware First
Before doing anything else, back up your current working firmware and note your GUIDs/MACs:
Code:
# All flint/mstflint/mlxlink commands require root/admin privileges
# Use sudo on Linux, run as Administrator on Windows
sudo flint -d mt4119_pciconf0 ri backup_fw.bin
sudo flint -d mt4119_pciconf0 query
Code:
# Option A: Patch existing firmware
python3 cx5_gen4_enable.py --input your_firmware.bin --output patched.bin
# Option B: Upgrade OEM to latest LTS + Gen4
python3 cx5_gen4_enable.py --input your_oem.bin --upgrade-base acat_8002.bin --output patched.bin
# Option C: Analyze firmware without patching
python3 cx5_gen4_enable.py --input your_firmware.bin --analyze
# Flash (FNP recovery mode needed for cross-vendor or signature errors)
sudo flint -d mt4119_pciconf0 -i patched.bin --skip_ci_req burn
# FULL power cycle (not reboot — the card needs a cold boot)
# Verify
sudo mlxlink -d mt4119_pciconf0
lspci -vvs <device> | grep -i "lnksta\|lnkcap"If you're flashing modified firmware, or flashing an image from a different vendor onto your card (e.g., Dell firmware onto an HPE card, or vice versa), the card's secure boot will reject the image because the signature no longer matches. The card needs to be in FNP (Firmware Not Present) recovery mode to bypass the signing check.
Common error messages that indicate you need FNP recovery mode:
Code:
-E- Burning FS4 image failed: The Digest in the signature is wrong
-E- MFE_DIRECT_MEM_ACCESS_DISABLED
-E- Flash access is disabled via the FW option. Use FNP jumper to recover.
-E- Cannot open device. Ensure secure boot is disabled or use FNP recovery.
-E- Secure boot is enabled on this deviceCurrent Status & Tested Hardware
| Card | Firmware | Gen4 | LEDs | Status |
|---|---|---|---|---|
| Dell 0TDNNT (CX512F) | Dell OEM 4554 + Gen4 patch | Working | Working | Primary test card |
| HPE P12608 (CX512F) | Dell OEM 4554 + Gen4 patch | Working | Working | Cross-vendor validated |
| Dell 0TDNNT (CX512F) | Dell-tuned ACAT 8002 + Gen4 | Working | Working | Latest LTS w/ OEM tuning |
| MCX555A-ECAT (VPI 1×QSFP28) | Stock 1012 + Gen4 patch | CRC valid | — | Pending bench test |
Port status LEDs function correctly on both tested cards. I've seen posts where people have had LED issues after firmware modifications — they're working as expected here on both the Dell and HPE hardware.
Community Help Needed
The tool should handle all CX5 variants but needs further testing on:
- QSFP28 cards (MCX555A, MCX515A, MCX516A) — the tool correctly patches only the 2 bytes needed, but hardware bench testing is still needed to confirm Gen4 trains.
- Single-port cards — tool now handles disabled port 2 automatically, needs hardware confirmation.
- Other OEM firmware images — Lenovo, Supermicro, Cisco UCS. The Gen4 patch offsets should be the same, but OEM profiles need to be built from firmware samples. If you have OEM CX5 firmware and the matching stock Mellanox version, that's enough to derive a profile.
- Older firmware versions (16.28.x, 16.32.x) — offsets are confirmed stable between 16.35.1012, 16.35.4554, and 16.35.8002, but earlier major versions haven't been checked.
Changelog
v1.1.0 — 2026-03-02
- Universal CX5 support — now handles EN, VPI, SFP28, QSFP28, single-port, and dual-port cards
- Fixed QSFP28 compatibility (MCX555A-ECAT) — QSFP28 cards use port_gen=0x06 for both Gen3 and Gen4, and speed tables ship pre-invalidated. Tool now recognizes these as valid and only patches the 2 bytes that need changing
- Added --analyze mode for firmware inspection without patching
- Added connector type and port count auto-detection
- Added backup and sudo/admin privilege reminders
- Initial release for CX5 EN dual-port SFP28 cards
- 8-byte Gen4 patch with native CRC-16
- Dell OEM profile and --upgrade-base mode
- Tested on Dell 0TDNNT, HPE P12608, stock ACAT
Acknowledgments
CRC-16 implementation derived from the mstflint open-source project (dual GPL-2.0/BSD license). Firmware analysis and tool development done with significant assistance from Claude. Tool and all documentation are MIT-licensed.
Edit: Accompanying Reddit post: Enabling PCIe Gen4 on ConnectX-5 EN Cards — Firmware Patch Tool
Last edited:
