Your results are close to mine, I could never get anywhere close to 40 regardless of what I did, but could hit the mid 20's on the regular.I couldn't quite "saturate" my 40GbE with any of my storage, but an M.2 or U.2 NVMe got me sustained speeds of 20Gbps+ direct connected without any special tuning via vDS (distributed switch). Did the quick math, that's about 2,500MB/s.
40GbE + might be useful for things like connecting machines with PCIe 4.0 NVMe or NVMe RAID setups. Beyond that, kinda overkill. On an all internal network my VMs could only hit 26-29Gbps to one another via iperf3 using VMXNET3 anyway. Not sure if that's a limitation of the linux kernel, or the X10SRL-F w/ E5-2650v4 I was using, I guess it's possible some MPPs-centric software like DPDK might be able to break that barrier, but haven't explored.
I just got the US-Pro-Aggregation for $550 on the fleabays. 4x25GbE backbone. They're coming down.But 25G switches are much harder to come by (ie much more expensive)
Yeah, it makes the 40 part of 40GbE pretty irrelevant. I guess it's nice to have the overhead if you "need" it, but it's just not necessary. Above 10Gbps is definitely beneficial, though, as mentioned by @i386 - which is why I'm pretty juiced about my new cards and switch, instead of sticking with QSFP+ and buying an old, crusty, loud AF power guzzler IB switch I have to fight with to even do ethernet.Your results are close to mine, I could never get anywhere close to 40 regardless of what I did, but could hit the mid 20's on the regular.
Used them couple years ago but their driver and firmware caused crash every time. Haven't checked after then, new firmware may have fixed that. I have 6-7 of these cards I'm just using 25G ethernet now.I'm very seriously weighing moving over to QL45212, any experience with those?
It took me less than 30 minutes and I'm a windows user who had to google how to make files in linux executable...I have to fight with to even do ethernet
What were you using the QL45212 with that crashed? And what do you mean "back to basics"?Used them couple years ago but their driver and firmware caused crash every time. Haven't checked after then, new firmware may have fixed that. I have 6-7 of these cards I'm just using 25G ethernet now.
I'm back to the basis now, keep it simple stupid. For home use, you are not going to see difference for these advanced features.
For an EMC SX3016? ... I haven't read it in years, but I saw one thread where people were struggling with it, was very long. Seemed harder than the process I followed for flashing MCX354A from 56Gb IB to 40GbE. But like I said, haven't read it in a long time. Don't need to now!It took me less than 30 minutes and I'm a windows user who had to google how to make files in linux executable...
chmod +x
homie. It's like, fleek. All the kids are doing it on the TikToks.roce, rdma. Back to NFS, simple and just works.What were you using the QL45212 with that crashed? And what do you mean "back to basics"?