School me on PCI-E bus speeds, GT/s, etc (X520 question inside!)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Brandon_K

New Member
Jan 17, 2021
26
17
3
Pittsburgh, PA
While laying out a new system, it's come to my realization that I don't have as much of an understanding on PCI-E bandwidth as I thought I did.

I'm looking at a Z690 based motherboard to build a new machine off of. It has 1 x16 PCIe 5.0, 2 x16 PCIe 3.0 and 1 x1 PCIe 3.0 slots (the x1 is not open ended). Unfortunately, the two x16 3.0 slots are limited to x1 and x4 according to the specs.

I have a X520-SR2 dual 10gbe (PCIe 2.0 x8), a IBM M5110 (LSI 2303, PCIe 3.0 x8) and a LSI 9207 (PCIe 3.0 x8) for this machine.

I'm having trouble wrapping my head around the difference between GT/s and GB/s. The 2.0 spec is 5GT/s per lane, which an online calculator shows to be 40GB/sec, but even a x16 can only do 8GB/sec throughput according to the table I'm looking at here. I'm confused at the correlation.

Ultimately I'm trying to figure out how hosed I am on bandwidth on the board. I *think* my best option is to put the M5110 in the x16 5.0 slot (internal drive array, fast disks), the X520 in the x16/x4 slot (I'd like to get the full 10gbe out of it) and the 9207-8e in the x16/x1 slot? I'm not thrilled at the idea, but the external drive shelfs house pretty slow 2.5" SMR drives, so at the end of the day it's not going to need a ton of bandwidth in the first place.

Guidance would be great :)
 

BlueFox

Legendary Member Spam Hunter Extraordinaire
Oct 26, 2015
2,063
1,482
113
T = transfer, B = byte. One transfer is essentially the equivalent of a bit, however, with encoding, there is overhead, so that's just raw throughput. With rounding, a single lane of PCIe is good for 250MB/s after overhead, doubling with each incremental version (so PCIe 3.0 is ~1GB/s per lane).

Your proposal seems reasonable and I can't think of a better configuration given your motherboard's layout, given that you want at least 1GB/s out of the NIC, which would require 2 lanes at minimum.
 
  • Like
Reactions: Brandon_K

NablaSquaredG

Layer 1 Magician
Aug 17, 2020
1,320
800
113
I'm having trouble wrapping my head around the difference between GT/s and GB/s. The 2.0 spec is 5GT/s per lane, which an online calculator shows to be 40GB/sec, but even a x16 can only do 8GB/sec throughput according to the table I'm looking at here. I'm confused at the correlation.
The online calculator is probably giving you the raw bit throughput in Gigabit/s, whereas the table on wikipedia gives the effective throughput (without protocol overhead etc) in Gigabyte/s

PCIe2.0 x8:
5GT/s * 8 lanes => 40 Gigabit/s raw bit throughput
40 Gigabit/s / 10 * 8 = 32 Gigabit/s = 4 Gigabyte/s (value from the table in wiki)

/10 * 8 comes from the 8b10b encoding (10 bits transmitted for 8 bits payload)


So
PCIe 5.0 x16 => 63.015 GB/s or 504.12 Gb/s (Gb = Gigabits per second)
PCIe 3.0 x4 => 3.938 GB/s or 31.504 Gb/s
PCIe 3.0 x1 => 0.985 GB/s or 7.88 Gb/s

So yeah, your options are limited.

The X520-SR2 in an 3.0 x4 slot will give you 16GBit/s max throughput (PCIe2.0 x4)
9207 in 3.0 x1 limits you to just below 1GByte/s
 
  • Like
Reactions: Brandon_K

i386

Well-Known Member
Mar 18, 2016
4,221
1,540
113
34
Germany
The online calculator is probably giving you the raw bit throughput in Gigabit/s, whereas the table on wikipedia gives the effective throughput (without protocol overhead etc) in Gigabyte/s

PCIe2.0 x8:
5GT/s * 8 lanes => 40 Gigabit/s raw bit throughput
40 Gigabit/s / 10 * 8 = 32 Gigabit/s = 4 Gigabyte/s (value from the table in wiki)

/10 * 8 comes from the 8b10b encoding (10 bits transmitted for 8 bits payload)


So
PCIe 5.0 x16 => 63.015 GB/s or 504.12 Gb/s (Gb = Gigabits per second)
PCIe 3.0 x4 => 3.938 GB/s or 31.504 Gb/s
PCIe 3.0 x1 => 0.985 GB/s or 7.88 Gb/s

So yeah, your options are limited.

The X520-SR2 in an 3.0 x4 slot will give you 16GBit/s max throughput (PCIe2.0 x4)
9207 in 3.0 x1 limits you to just below 1GByte/s
You were faster than me :D
 

NateS

Active Member
Apr 19, 2021
159
91
28
Sacramento, CA, US
How committed are you to using that particular network card? If you used one that supported PCIe3, it would have full bandwidth in your 3.0x4 slot, and your M5110 would have full bandwidth in the PCIe5x16 slot, and only your external spinners would be bandwidth limited by the slot. That would probably be the cheapest way to have full bandwidth for both your fast disks and network card.
 

Brandon_K

New Member
Jan 17, 2021
26
17
3
Pittsburgh, PA
How committed are you to using that particular network card? If you used one that supported PCIe3, it would have full bandwidth in your 3.0x4 slot, and your M5110 would have full bandwidth in the PCIe5x16 slot, and only your external spinners would be bandwidth limited by the slot. That would probably be the cheapest way to have full bandwidth for both your fast disks and network card.
I don't need to use both ports of the NIC, so it's not particularly bandwidth limited in the x4 slot. If I'm doing all of the math correctly, the NIC will end up with 16gbps, the M5110 will have a full 63gbps and the 9207 will end up with 7.88gbps.

That 9207 connects to a DS2246, filled with (24) 5TB spinners. They max out at 135MB/sec, so I would have to have 7 drives spinning at the same time to max out the PCI bus, which just isn't going to happen. The array rarely gets written to, only read from. Most everything 'working' data wise will be on the four NVME drives on the motherboard.

Ultimately I'm going to keep my eye out for a 9305-16i. I intend on upgrading the backplane in my 826 to have more NVME drives (the latest backplane has 4xNvme on the back), which is 12gbps. I can get a bracket to pass one of the ports off of the 9305 to a 8088 for the DS2466.

Thanks for all of the help guys.
 
Last edited:

nabsltd

Active Member
Jan 26, 2022
340
208
43
I'm looking at a Z690 based motherboard to build a new machine off of.
Pretty much every LGA 1700 motherboard I have looked at is designed with only the 20 PCIe lanes from the CPU being routed to PCIe slots. The 8x DMI that the chip supports and connects it to the chipset (like the Z690) has more than enough bandwidth to handle 4x M.2 slots and still have some PCIe slots routed through it, but nobody is building that way right now.

I dropped back a generation to get 48x PCIe lanes on the chip, and have a huge choice of boards with lots of PCIe slots. But, I paid the penalty of having much lower CPU performance compared to the Alder Lake offerings.

You always need to start by looking at which is more important: I/O bandwidth or CPU speed, since Intel's tick/tock seem to alternate between the two.
 

BlueFox

Legendary Member Spam Hunter Extraordinaire
Oct 26, 2015
2,063
1,482
113
You always need to start by looking at which is more important: I/O bandwidth or CPU speed, since Intel's tick/tock seem to alternate between the two.
It's not due to Intel's model, it's entirely different market segments. The consumer products have more PCIe lanes than ever before. The HEDT offerings were based on the mainstream server CPUs, which have always had more PCIe lanes, but are not revised as frequently. Popularity on those declined when single-threaded performance on the consumer line was much higher and fewer systems had multiple GPUs. They're pretty niche setups now.
 

RageBone

Active Member
Jul 11, 2017
617
159
43
I find that usually, Gigabit is shortened with Gb and GigaByte with GB.
Gb divided by 8 = GB

And then there is Gibi ...