Single Port 25gbe NIC's with 4x PCIe Gen3?

[mattlach]

Hey everyone,

I'm in a bit of a bind, trying to ghetto-mod some extra network capacity into a mini-ITX system and only have a 4x m.2 slot available to me.

I'm considering using a m.2 to 4x PCIe slot riserand creatively mounting it somehow.

Ideally I'd want 25gbe bandwidth out of this solution.

Does anyone know if there are any single slot NIC's that work at full speed on 4x PCIe? Gen 3 or even Gen 4 would do the trick.

I've seen a few dual port ones with 8x slots, but in my experience just running these with half the number of PCIe lanes and only using one port doesnt work well, at least judging by the old Intel AT2 adapters I've tried this on.

Worse comes to worse, I'll just do 10gbe, but I'd really like more bandwidth on this solution.

Appreciate any suggestions!

--Matt

 

[mattlach]

...or does anyone have any experience with any brand of 25Gbit adapter, even dual port ones, working well and attaining full speeds when plugged into an x4 gen3/4 port, despite being an x8 card, if only one of the ports were in use?

Just because my decade old Intel adapters didn't do this, doesn't mean that no new ones will.

 

[chenxiaolong]

I'm doing a similar thing with a dual port Mellanox ConnectX-4 card, although it's with 40Gbps optics and it's an x16 card in an x8 slot. I have no problems maxing out the link with a single port.

 

[slushy]

I'm doing a similar thing with a dual port Mellanox ConnectX-4 card, although it's with 40Gbps optics and it's an x16 card in an x8 slot. I have no problems maxing out the link with a single port.

how did you achieve this? i have it connected to a pcie 4.0 4x and it maxes out at half bandwidth

 

[Matta]

how did you achieve this? i have it connected to a pcie 4.0 4x and it maxes out at half bandwidth

ConnectX-4 is Pcie 3.0 card. When you connect it to Pcie 4.0 x4 you still get pcie 3.0 x4 bandwidth.

 

[slushy]

ConnectX-4 is Pcie 3.0 card. When you connect it to Pcie 4.0 x4 you still get pcie 3.0 x4 bandwidth.

true which is about 32gbe if i am not mistaken, but thus far, the connectx4 i have under my possession runs at half bandwidth under 4x, i.e. 5gb on 10gbe link, 10~ish on 25gbe link ethernet mode. Did you have to change anything?

 

[bugacha]

true which is about 32gbe if i am not mistaken, but thus far, the connectx4 i have under my possession runs at half bandwidth under 4x, i.e. 5gb on 10gbe link, 10~ish on 25gbe link ethernet mode. Did you have to change anything?

What does it mean 10ish on 25gbe link?

How do you measure this? I hope you use -P 4 with iperf3

 

[pimposh]

PCI3x8card (eg. CX4) will do around 22,5Gbps of real traffic when linked at PCI3x4.
Overheads of different nature are limiters here.