RDMA and CX4LX Troubleshooting

gerby · Jan 23, 2026

Running 2 CX4LXs in Proxmox and all my GIDs have disappeared. I'm at a loss, just tried installing the latest DOCA package and DKMS modules; firmware is up to date.

Code:

# show_gids
DEV     PORT    INDEX   GID                                     IPv4            VER     DEV
---     ----    -----   ---                                     ------------    ---     ---
n_gids_found=0

Code:

# ibstatus | grep -b1 gid
0-Infiniband device 'mlx5_0' port 1 status:
42:     default gid:     unknown
65-     base lid:        0x0
--
192-Infiniband device 'mlx5_0' port 10 status:
235:    default gid:     unknown
258-    base lid:        0x0
--
385-Infiniband device 'mlx5_0' port 2 status:
427:    default gid:     unknown
450-    base lid:        0x0
--
577-Infiniband device 'mlx5_0' port 3 status:
619:    default gid:     unknown
642-    base lid:        0x0
--
769-Infiniband device 'mlx5_0' port 4 status:
811:    default gid:     unknown
834-    base lid:        0x0
--
961-Infiniband device 'mlx5_0' port 5 status:
1003:   default gid:     unknown
1026-   base lid:        0x0
--
1153-Infiniband device 'mlx5_0' port 6 status:
1195:   default gid:     unknown
1218-   base lid:        0x0
--
1345-Infiniband device 'mlx5_0' port 7 status:
1387:   default gid:     unknown
1410-   base lid:        0x0
--
1537-Infiniband device 'mlx5_0' port 8 status:
1579:   default gid:     unknown
1602-   base lid:        0x0
--
1729-Infiniband device 'mlx5_0' port 9 status:
1771:   default gid:     unknown
1794-   base lid:        0x0
--
1921-Infiniband device 'mlx5_1' port 1 status:
1963:   default gid:     unknown
1986-   base lid:        0x0
--
2113-Infiniband device 'mlx5_1' port 10 status:
2156:   default gid:     unknown
2179-   base lid:        0x0
--
2306-Infiniband device 'mlx5_1' port 11 status:
2349:   default gid:     unknown
2372-   base lid:        0x0
--

--
2499-Infiniband device 'mlx5_1' port 12 status:
2542:   default gid:     unknown
2565-   base lid:        0x0
--
2692-Infiniband device 'mlx5_1' port 13 status:
2735:   default gid:     unknown
2758-   base lid:        0x0
--
2885-Infiniband device 'mlx5_1' port 14 status:
2928:   default gid:     unknown
2951-   base lid:        0x0
--
3078-Infiniband device 'mlx5_1' port 15 status:
3121:   default gid:     unknown
3144-   base lid:        0x0
--
3271-Infiniband device 'mlx5_1' port 16 status:
3314:   default gid:     unknown
3337-   base lid:        0x0
--
3464-Infiniband device 'mlx5_1' port 17 status:
3507:   default gid:     unknown
3530-   base lid:        0x0
--
3657-Infiniband device 'mlx5_1' port 18 status:
3700:   default gid:     unknown
3723-   base lid:        0x0
--
3850-Infiniband device 'mlx5_1' port 2 status:
3892:   default gid:     unknown
3915-   base lid:        0x0
--
4042-Infiniband device 'mlx5_1' port 3 status:
4084:   default gid:     unknown
4107-   base lid:        0x0
--
4234-Infiniband device 'mlx5_1' port 4 status:
4276:   default gid:     unknown
4299-   base lid:        0x0
--
4426-Infiniband device 'mlx5_1' port 5 status:
4468:   default gid:     unknown
4491-   base lid:        0x0
--
4618-Infiniband device 'mlx5_1' port 6 status:
4660:   default gid:     unknown
4683-   base lid:        0x0
--
4810-Infiniband device 'mlx5_1' port 7 status:
4852:   default gid:     unknown
4875-   base lid:        0x0
--
5002-Infiniband device 'mlx5_1' port 8 status:
5044:   default gid:     unknown
5067-   base lid:        0x0
--
5194-Infiniband device 'mlx5_1' port 9 status:
5236:   default gid:     unknown
5259-   base lid:        0x0
--
5386-Infiniband device 'mlx5_2' port 1 status:
5428:   default gid:     unknown
5451-   base lid:        0x0

Code:

# mlxfwmanager
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX4LX
  Part Number:      MCX4111A-ACA_Ax
  Description:      ConnectX-4 Lx EN network interface card; 25GbE single-port SFP28; PCIe3.0 x8; ROHS R6
  PSID:             MT_2410110034
  PCI Device Name:  0000:25:00.0
  Base MAC:         0c42a1b9ed06
  Versions:         Current        Available
     FW             14.32.1908     N/A
     PXE            3.6.0502       N/A
     UEFI           14.25.0017     N/A

  Status:           No matching image found

Device #2:
----------

  Device Type:      ConnectX4LX
  Part Number:      MCX4111A-ACA_Ax
  Description:      ConnectX-4 Lx EN network interface card; 25GbE single-port SFP28; PCIe3.0 x8; ROHS R6
  PSID:             MT_2410110034
  PCI Device Name:  0000:63:00.0
  Base MAC:         0c42a1ad7186
  Versions:         Current        Available
     FW             14.32.1908     N/A
     PXE            3.6.0502       N/A
     UEFI           14.25.0017     N/A

  Status:           No matching image found

Anyone have any hints or ideas here? I'm at a loss as to even where to start troubleshooting.

dsrhdev · Jan 24, 2026

hello,
what does [mst]flint show?

Code:

⬢ [root@t-251 roothome]# mstflint -d mlx4_0 q
Image type:            FS2
FW Version:            2.42.5000
FW Release Date:       5.9.2017
Product Version:       02.42.50.00
Rom Info:              type=PXE version=3.4.752
Device ID:             4099
Description:           Node             Port1            Port2            Sys image
GUIDs:                 f4521403007f0f40 f4521403007f0f41 f4521403007f0f42 f4521403007f0f43
MACs:                                       f452147f0f41     f452147f0f42
VSD:                 
PSID:                  MT_1090120019

you also can restore guids by macs, i had the similar problem when was reflashing cx3 pro en to vpi fw

gerby · Jan 24, 2026

Code:

for i in 0 1
> do
> mstflint -d mlx5_${i} q
> done
Image type:            FS3
FW Version:            14.32.1908
FW Release Date:       17.8.2025
Product Version:       14.32.1908
Rom Info:              type=UEFI version=14.25.17 cpu=AMD64,AARCH64
                       type=PXE version=3.6.502 cpu=AMD64
Description:           UID                GuidsNumber
Base GUID:             0c42a10300b9ed06        4
Base MAC:              0c42a1b9ed06            4
Image VSD:             N/A
Device VSD:            N/A
PSID:                  MT_2410110034
Security Attributes:   N/A
Image type:            FS3
FW Version:            14.32.1908
FW Release Date:       17.8.2025
Product Version:       14.32.1908
Rom Info:              type=UEFI version=14.25.17 cpu=AMD64,AARCH64
                       type=PXE version=3.6.502 cpu=AMD64
Description:           UID                GuidsNumber
Base GUID:             0c42a10300ad7186        4
Base MAC:              0c42a1ad7186            4
Image VSD:             N/A
Device VSD:            N/A
PSID:                  MT_2410110034
Security Attributes:   N/A

dsrhdev · Jan 24, 2026

gerby said:

Code:

for i in 0 1
> do
> mstflint -d mlx5_${i} q
> done
Image type:            FS3
FW Version:            14.32.1908
FW Release Date:       17.8.2025
Product Version:       14.32.1908
Rom Info:              type=UEFI version=14.25.17 cpu=AMD64,AARCH64
                       type=PXE version=3.6.502 cpu=AMD64
Description:           UID                GuidsNumber
Base GUID:             0c42a10300b9ed06        4
Base MAC:              0c42a1b9ed06            4
Image VSD:             N/A
Device VSD:            N/A
PSID:                  MT_2410110034
Security Attributes:   N/A
Image type:            FS3
FW Version:            14.32.1908
FW Release Date:       17.8.2025
Product Version:       14.32.1908
Rom Info:              type=UEFI version=14.25.17 cpu=AMD64,AARCH64
                       type=PXE version=3.6.502 cpu=AMD64
Description:           UID                GuidsNumber
Base GUID:             0c42a10300ad7186        4
Base MAC:              0c42a1ad7186            4
Image VSD:             N/A
Device VSD:            N/A
PSID:                  MT_2410110034
Security Attributes:   N/A

ok, you have both base guid&mac defined
so the problem might be about loaded modules, like

Code:

# lsmod | grep 'mlx5_ib\|uverb'
mlx5_ib               487424  0
ib_uverbs             200704  2 mlx4_ib,mlx5_ib
ib_core               516096  10 isert_scst,rdma_cm,ib_ipoib,rpcrdma,mlx4_ib,iw_cm,ib_isert,ib_uverbs,mlx5_ib,ib_cm
mlx5_core            2138112  1 mlx5_ib

if they are loaded than `ibstat' should work ok

Code:

# ibstat
CA 'mlx5_0'
        CA type: MT4115
        Number of ports: 1
        Firmware version: 12.28.2302
        Hardware version: 0
        Node GUID: 0x0c42a1030060919a
        System image GUID: 0x0c42a1030060919a    
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 100
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x0e42a1fffe60919a
                Link layer: Ethernet
CA 'mlx5_1'
        CA type: MT4115
        Number of ports: 1
        Firmware version: 12.28.2302
        Hardware version: 0
        Node GUID: 0x0c42a1030060919b
        System image GUID: 0x0c42a1030060919a    
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 7
                LMC: 0
                SM lid: 1
                Capability mask: 0x2651e848
                Port GUID: 0x0c42a1030060919b
                Link layer: InfiniBand

in my case i have dual-port vpi adapter, where port1 is in an ethernet mode, and port2 is in infiniband mode,
you should see two ethernet mode ports (it might needs to be loaded the `ib_umad' kernel module)

gerby · Jan 24, 2026

I'm at a loss.

Code:

# lsmod | grep 'mlx5_ib\|uverb'
mlx5_ib               540672  0
ib_uverbs             200704  2 rdma_ucm,mlx5_ib
macsec                 69632  1 mlx5_ib
ib_core               507904  9 rdma_cm,ib_ipoib,iw_cm,ib_umad,ib_isert,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
mlx5_core            3035136  2 mlx5_fwctl,mlx5_ib
mlx_compat             16384  18 rdma_cm,ib_ipoib,mlxdevm,nvme_keyring,nvme,mlxfw,mlx5_fwctl,iw_cm,nvme_core,nvme_fabrics,ib_umad,ib_isert,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core

gerby · Jan 24, 2026

Code:

# ibstat
ibwarn: [2583199] main: stat of IB device 'mlx5_0' failed
ibwarn: [2583199] main: stat of IB device 'mlx5_1' failed
ibwarn: [2583199] main: stat of IB device 'mlx5_2' failed

dsrhdev · Jan 24, 2026

gerby said:

Code:

# ibstat
ibwarn: [2583199] main: stat of IB device 'mlx5_0' failed
ibwarn: [2583199] main: stat of IB device 'mlx5_1' failed
ibwarn: [2583199] main: stat of IB device 'mlx5_2' failed

are there any errors in dmesg (afrter reboot)?
dmesg | grep 'mlx5\|DMAR'

if you have virtualization enabled, like this

Code:

intel_iommu=on iommu=pt pci=realloc intremap=no_x2apic_optout

please check you also have "Enable 4G encoding' in BIOS

dsrhdev · Jan 25, 2026

just plug cx4lx

it looks like you have some kind of library/dependencies problem, did you try to boot your favorite (live) distro and check the cards?

gerby · Jan 30, 2026

dsrhdev said:
are there any errors in dmesg (afrter reboot)?
dmesg | grep 'mlx5\|DMAR'

if you have virtualization enabled, like this

Code:

intel_iommu=on iommu=pt pci=realloc intremap=no_x2apic_optout

please check you also have "Enable 4G encoding' in BIOS

No errors in dmseg that I can find. Kernel cmdline is as follows:

Code:

BOOT_IMAGE=/boot/vmlinuz-6.17.4-2-pve root=/dev/mapper/pve-root ro idle=nomwait quiet amd_iommu=on iommu=pt mem_encrypt=on irqpoll default_hugepagesz=1G hugepagesz=1G hugepages=16

gerby · Jan 30, 2026

No messages at all in dmesg from DMAR, after a boot mlx5 looks fine...

gerby · Jan 30, 2026

Sometimes it takes looking at your config in your own post on a forum to see things. Conceptually I'd _expect_

Code:

mem_encrypt=on

to break RDMA; but it never even crossed my mind till I was staring at it here. Disabling and testing; will report back.

gerby · Jan 30, 2026

Nope, same result

ㄟ( ▔, ▔ )ㄏ

dsrhdev · Jan 31, 2026

hello,
it looks like something wrong with infiniband libs/deps

Search

RDMA and CX4LX Troubleshooting

gerby

SREious Engineer

dsrhdev

Member

gerby

SREious Engineer

dsrhdev

Member

gerby

SREious Engineer

gerby

SREious Engineer

dsrhdev

Member

dsrhdev

Member

Attachments

gerby

SREious Engineer

gerby

SREious Engineer

gerby

SREious Engineer

gerby

SREious Engineer

dsrhdev

Member