What is the Xeon V4 CPU to CPU bandwidth?

[voodooFX]

I need a server with 2x Mellanox EDR Infiniband ports (100 Gbit/s)

Since a single PCIe 16X Gen 3.0 can handle up to 15.754 GB/s and a single EDR port needs 12.5 GB/s
it's clear that a dual port card is not a suitable solution.

If I use two EDR cards (in a dual socket system), each one connected to a different CPU, I would be limited by the CPU (QPI) bandwidth in a passthrough scenario? (one card reading and the other one writing?)

Briefly: what is the QPI bandwidth in Xeon V4 architecture?

 

[aero]

graph from Detailed Specifications of the Intel Xeon E5-2600v4 "Broadwell-EP" Processors | Microway

As you can see, there are models with 8GT/s QPI, and some with 9.6GT/s. That yields effective bandwidth of 128Gb/s and 153.6Gb/s uni-directionally, respectively.

edit: it's the same as the haswell v3's

 

[voodooFX]

So 16GB/s for cheaper models and 19.2GB/s for the others
I'm wondering how much the performances would be affected by the QPI, even if the bandwidth is actually more than enough, assuming there is no concurrent traffic...

 

[vanfawx]

If you don't mind, what are you building that needs so much bandwidth?

 

[voodooFX]

I work in a supercomputing centre
We avoided the QPI (possible) problem installing the EDR cards in two PCIe slots controlled by the same CPU