Processor/RAM limitations on 40Gbe?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

darkconz

Member
Jun 6, 2013
193
15
18
I am wondering if there is a minimum requirement to get up to 40 Gbe networking? I noticed something being very slow with my s2d setup and I was testing all sorts of things. I was able to confirm RDMA is working but when I was testing my network iperf, it seems like it would not go past 8 Gbe.

All the nodes are:
2x E5-2630L
32GB DDR3 2133MHz
Dual port ConnectX-3 with SET binding the two ports together
Power setting is set to high

I read that the processor and ram could be the bottleneck but I remember when I first set this up 2 years ago, the speeds were near 40Gbe before deployment. I've been scratching my head for the past week trying to nail this down but I just couldn't...

Any pointer is appreciated. Thanks!
 

oddball

Active Member
May 18, 2018
206
121
43
42
v3 or v4 CPU? I'm guessing so based on the RAM. The CPU should handle it without an issue.

What OS? In Windows I've had trouble getting past 32-33GbE with all the tweaked settings. All of our Linux boxes are on dual 20GbE connections, but out of the box untweaked we can saturate a 20GbE connection on Linux, that's much harder on Windows.

What switch are you running this through?
 

darkconz

Member
Jun 6, 2013
193
15
18
Those are only V2 CPUs. I am doing tests on Windows, haven't done it in Linux.

These are going through Arista DCS-7050QX-32.

Funny thing is, I am 100% positive I've done these tests before putting these into production. I couldn't find the notes but if the system was 1/4 of the rated capacity, I would have done more investigation prior to putting into production. For some odd reason, I was noticing slow transfer speeds via SMB recently which prompted me to retest these again.
 

kapone

Well-Known Member
May 23, 2015
1,095
642
113
As a datapoint - With Windows Server 2016 (and 2019) and Mellanox CX3 Pro 40gb cards and using RDMA, to reach 40gb speeds took up ~75% of a quad core 3.5GHz CPU. Amount of RAM did not seem to make much of a difference, but memory bandwidth did. I was using dual channel 1866 DDR3 RAM, but I suspect with quad channel it may do fine even with slower RAM.