Has anyone else "downgraded" from QSFP+ (40GbE) to SFP+ (10GbE) gear? And if so, why?

[Rand__]

But 25G switches are much harder to come by (ie much more expensive)

 

[Joshh]

Yeah, we are in this middle gap where 40gbe is relatively cheap, but going beyond that (in terms of newer tech) 25, 50, 100 is expensive.

 

[Joshh]

I couldn't quite "saturate" my 40GbE with any of my storage, but an M.2 or U.2 NVMe got me sustained speeds of 20Gbps+ direct connected without any special tuning via vDS (distributed switch). Did the quick math, that's about 2,500MB/s.

40GbE + might be useful for things like connecting machines with PCIe 4.0 NVMe or NVMe RAID setups. Beyond that, kinda overkill. On an all internal network my VMs could only hit 26-29Gbps to one another via iperf3 using VMXNET3 anyway. Not sure if that's a limitation of the linux kernel, or the X10SRL-F w/ E5-2650v4 I was using, I guess it's possible some MPPs-centric software like DPDK might be able to break that barrier, but haven't explored.

Your results are close to mine, I could never get anywhere close to 40 regardless of what I did, but could hit the mid 20's on the regular.

 

[AveryFreeman]

But 25G switches are much harder to come by (ie much more expensive)

I just got the US-Pro-Aggregation for $550 on the fleabays. 4x25GbE backbone. They're coming down.

The only thing that irks me is using the backbone part of the switch for my servers again, just like my 1Gbps switches with 4x10GbE SFP+. But presumably I can do a 4-port 10GbE LAGG between the two switches now and still have 24 open 10GbE SFP+ ports left to play with.

 

[AveryFreeman]

Your results are close to mine, I could never get anywhere close to 40 regardless of what I did, but could hit the mid 20's on the regular.

Yeah, it makes the 40 part of 40GbE pretty irrelevant. I guess it's nice to have the overhead if you "need" it, but it's just not necessary. Above 10Gbps is definitely beneficial, though, as mentioned by @i386 - which is why I'm pretty juiced about my new cards and switch, instead of sticking with QSFP+ and buying an old, crusty, loud AF power guzzler IB switch I have to fight with to even do ethernet.

Anyone want to buy a MCX354A-XCCT?

 

[vangoose]

I'm very seriously weighing moving over to QL45212, any experience with those?

Used them couple years ago but their driver and firmware caused crash every time. Haven't checked after then, new firmware may have fixed that. I have 6-7 of these cards I'm just using 25G ethernet now.

I'm back to the basis now, keep it simple stupid. For home use, you are not going to see difference for these advanced features.

 

[i386]

I have to fight with to even do ethernet

It took me less than 30 minutes and I'm a windows user who had to google how to make files in linux executable...

 

[AveryFreeman]

Used them couple years ago but their driver and firmware caused crash every time. Haven't checked after then, new firmware may have fixed that. I have 6-7 of these cards I'm just using 25G ethernet now.

I'm back to the basis now, keep it simple stupid. For home use, you are not going to see difference for these advanced features.

What were you using the QL45212 with that crashed? And what do you mean "back to basics"?

 

[AveryFreeman]

It took me less than 30 minutes and I'm a windows user who had to google how to make files in linux executable...

For an EMC SX3016? ... I haven't read it in years, but I saw one thread where people were struggling with it, was very long. Seemed harder than the process I followed for flashing MCX354A from 56Gb IB to 40GbE. But like I said, haven't read it in a long time. Don't need to now!

chmod +x homie. It's like, fleek. All the kids are doing it on the TikToks.

 

[vangoose]

What were you using the QL45212 with that crashed? And what do you mean "back to basics"?

roce, rdma. Back to NFS, simple and just works.