I have some further info regarding Windows 11 and 10gb networking. Might shed some further light on the issues OP ran into.
I was working on one of my older test motherboards tonight, an Asrock Z170 Extreme4. Its running a I7-6700K, and 16 gigs of ram. I was testing to see if the Intel X520 , X710 and E810 cards would behave performance wise if you only give it 4x pci-e lanes. Long story short, they did fine with 4x lanes in the first tested motherboard. I was using Iperf3 3.16 for all my tests.
Initially I was using Debian 12 to run all my tests. After I finished up in Debian, I decided to try Windows 10 LTSC 21H2, Windows server 2022, and Windows 11 23H2 pro to see how the various Windows OS's compared. I had previously tested the cards on my more powerful test machine (a I7-9700K on a Z370 motherboard), but the Z370 box has a different pci-e configuration and I wanted to make sure I was only giving the cards 4x.
Windows 10 performed as expected, giving a solid 9.5gb in Iperf with all the cards. Server 2022 also behaved the same. Windows 11 though......
The results with 11 were all over the place, and no card, even the vaunted Mellanox 4x, was able to give consistent network transfers. I even moved the cards to the 16x lane slot to see if that would help, and shockingly it made things even worse!
I wondered if this was some sort of bug with this old board, so I dug out another old test board an Asrock Z170 pro4. The pro4 was even worse! It could barely hold 5gb, regardless of what card I tried, or what slot I plugged the card into, or even what OS was tested.
Conclusion #1 - I think the server grade cards we're all using really can get PICKY about all sorts of things. I don't think the vendors really test or design for consumer grade systems. Different motherboards, different OS's, different pci-e slots, etc. Change one variable, and you can get totally different results.
Conclusion #2 - Windows 11 was the most temperamental OS in terms of network performance. On my Z370 it performs fine, but not on the Z170 boards (either one). Considering how M$ has treated 11 as a non stop beta I'm not terribly surprised though.
Conclusion #3 - The Mellanox 4x's seem to be the most "stable" in terms of network performance. But even they're not immune to buggy motherboards (i.e my Asrock pro 4 ), or certain combinations of things (i.e. Win11 + the Extreme4 Z170).
Conclusion #4 - Keep a Linux box with a modern distro handy(or at least a flash drive with a live OS). It makes ruling out weirdness in Windows kernel or drivers, or the Windows iperf3 client a lot easier.
Conclusion #5 - consumer motherboards has unknown bugs or quirks in them that can cause issues. I never would of expected my pro 4 test board to act the way it did, yet the extreme4 board behaved just fine.