Stumped by 10Gb speed limit on 40/56gb IB cards...

[MichalPL]

I don't know what is the practical limit on PCIE 2.0 x8, but I am guessing around 3.2-3.6GB/s, you achieved already 3.1GB/s yes to max 40GbE to the limit PCIE 3.0 is needed. Maybe somebody test it already ? New PCIE 3.0 platform should be slightly faster.

 

[Sniper_X]

Yep, you’re right.
I have another platform I can use.
It’s a Ford note cluster that I intended the system to operate in eventually anyway.

I’ll go ahead and move my cards and testing over to those four notes and repeat my tests.

Will update as soon as I can.

 

[i386]

I don't know what is the practical limit on PCIE 2.0 x8, but I am guessing around 3.2-3.6GB/s, you achieved already 3.1GB/s yes to max 40GbE to the limit PCIE 3.0 is needed. Maybe somebody test it already ? New PCIE 3.0 platform should be slightly faster.

And don't forget the overhead of qdr and ipoib.

 

[Sniper_X]

Welp! thats the issue..

Look at the bandwidth on PCIe 2.0 x8:

 

[zer0sum]

Welp! thats the issue..

Look at the bandwidth on PCIe 2.0 x8:
View attachment 17916

Is that image correct?
I thought the per lane bandwidth was as follows:

• PCIe 2.0 = 5GT/s or 500MB/s
• PCIe 3.0 = 8GT/s or 1GB/s
• PCIe 4.0 = 16GT/s or 2GB/s

PCIe 2.0 is 500MB/s per lane bidirectional, and if you have 8 lanes of pcie 2.0 that gives you 4000MB/s.
Then of course there is some overhead

 

[Sniper_X]

PCIe 2.0 is 500MB/s per lane bidirectional, and if you have 8 lanes of pcie 2.0 that gives you 4000MB/s.
Then of course there is some overhead

Yes, overhead.
Plus Windows is a pig, so i'm moving tto a system that is PCIe 3.0.

Testing soon.