Mellanox ConnectX 4 and Windows 11 and Server 2022 lots of problems :-(

[Lance Longreen]

Hello folks.

I have some major issues. Have 8 units of the Mellanox CX4 dual 100 gig cards that i used before and they work in Truenas without issues. I have used them before on Windows 10 without problems. Now i tried to use them In Win11 Workstation on 3 different systems, 1 is a TR Pro 5000 on Asus WRX80, another is Asrock Rack Genoa and 3rd is Asrock Rack Rome (Milan CPU) and on 2 of the systems Windows 11 crashes every single reboot if i add the MX4 NIC and on the 3rd system it can boot into Windows 11 but i get Device cant start Error code (10) in device manager

Edit: I also now tried on a Xeon Skylake Workstation and its same issue. Since i have so many cards and they do work in Truenas core and Truenas Scale, i am a bit at my wits end here..When i google the code i see some people said they had similar issues but it never has resolution.

Hope some of the brains here can help in this matter.

All run latest firmware from Mellanox

 

[DavidWJohnston]

If this helps:

I run an MX4 single 100G (Branded as HPE 840QSFP28 but it's VEN_15B3&DEV_1013) in Win11 Pro WS in a Prime Z270-A, with RDMA (RoCE) and it works perfectly. I've never experienced the symptoms you describe. Details:

Firmware: 12.28.2006
Driver: 3.0.25668.0
Driver Date: 2022-07-20

I also run an MX4 dual 50G (VEN_15B3&DEV_1015) in ESXi as PCI-Passthrough to Windows Server 2022 DC, which works fine. Details:

Firmware: 14.24.1000
Driver: 3.20.25915.0
Driver Date: 2023-01-18

You could try changing the PCIe Boot ROM (aka Option ROM) to disabled, BIOS, or UEFI. Sometimes those prevent POST but I don't think it would prevent the device from starting if it boots successfully.

 

[Lance Longreen]

If this helps:

I run an MX4 single 100G (Branded as HPE 840QSFP28 but it's VEN_15B3&DEV_1013) in Win11 Pro WS in a Prime Z270-A, with RDMA (RoCE) and it works perfectly. I've never experienced the symptoms you describe. Details:

Firmware: 12.28.2006
Driver: 3.0.25668.0
Driver Date: 2022-07-20

I also run an MX4 dual 50G (VEN_15B3&DEV_1015) in ESXi as PCI-Passthrough to Windows Server 2022 DC, which works fine. Details:

Firmware: 14.24.1000
Driver: 3.20.25915.0
Driver Date: 2023-01-18

You could try changing the PCIe Boot ROM (aka Option ROM) to disabled, BIOS, or UEFI. Sometimes those prevent POST but I don't think it would prevent the device from starting if it boots successfully.

Appreciate it, tried boot rom order etc and resize BAR etc etc still same issues :-(