Diagnosing "slower" PCIe 4.0 SSD speeds on Epyc

[autoturk]

Hi all,

I'm having some puzzling behavior with u.2 PCIe 4.0 drives on an epyc platform I just built. I am testing a Intel/Solidigm 7.68tb p5500 u.2 drive. I have two systems:

Systems

Epyc
Epyc 7702p 64 core
Tyan S8030 motherboard
64 GB ddr4-3200 (octo-channel)

Alder Lake (AL)
i7 12700k
Asrock Rack W680d4u-blahblah motherboard
64 GB ddr5 4800

Drive is connected via Slimsas using riser at top PCIe slot on both systems.

Everything else is otherwise the same, running on ubuntu 20.04 on a separate NVMe SSD, using kdiskmark to benchmark the drives.

Problem

The odd thing is that I get slower reads and writes on the Epyc system. Specifically, I'm getting about 1 GB/s slower sequential reads, and much slower 4k Random writes (ie. last row, second column), though subsequent runs of the benchmark show at times even worse performance. In addition, when running 100 iterations of Ubuntu's built-in disk benchmark, I'm getting lots of random dips in read performance. The AL platform (even when trying to run the drive through an Oculink connector through the PCH lanes instead of the CPU) is rock solid and stable performance. Here are screenshots of the benchmarks.

Things I've tried:
- looking for PCIe errors in `dmesg`, or syslog and found nothing, though the Tyan board doesn't seem to have an AER option in the BIOS.
- Using the onboard Slimsas connector instead of a riser -- no change.
- Used a U.2 carrier card - no change.
- Loading optimized defaults in BIOS - no change.
- Updated BIOS - no change.

Things I've yet to try:
- Trying the drive in a Ryzen 7000 machine.
- Using an Oculink riser in the top slot.


Any thoughts? This is not a huge deal in the grand scheme of things but I thought it might be indicative of something more problematic, especially those odd dips/inconsistency in the ubuntu built-in benchmark.

 

[RolloZ170]

Alder may be faster with only one drive, but with a dozen EPYCs will win.

 

[autoturk]

Alder may be faster with only one drive, but with a dozen EPYCs will win.

well I've tried both systems with up to three drives and hit them all simultaneously, and same thing again. Are you suggesting it's an issue with the lower single threaded performance of Epyc?

 

[RolloZ170]

Are you suggesting it's an issue with the lower single threaded performance of Epyc?

lower single threaded performance is one reason.
at EPYCs the way from core to PCIe lane is more complicated. some lanes can be far from the core the code is running on.
will be nice of you direct the code to selected cores to find out a difference.

 

[autoturk]

lower single threaded performance is one reason.
at EPYCs the way from core to PCIe lane is more complicated. some lanes can be far from the core the code is running on.
will be nice of you direct the code to selected cores to find out a difference.

oh good point. Any straightforward ways of implementing your suggestion of trying out different cores?

 

[RolloZ170]

Any straightforward ways of implementing your suggestion of trying out different cores?

on windows yes. linux: it is not possible, i have no idea how, ask a linux specialist.
another part in the chain: what about the NVMe driver ?

 

[autoturk]

on windows yes. linux: it is not possible, i have no idea how, ask a linux specialist.
another part in the chain: what about the NVMe driver ?

I'm happy to try this out in windows (in fact I did the benchmarks in windows too, and got similar results). Can you point me to something?

The driver should be consistent since I was using exactly the same Ubuntu installation. I've also updated the firmware on all drives as well

 

[RolloZ170]

The driver should be consistent since I was using exactly the same Ubuntu installation. I've also updated the firmware on all drives as well

AMD driver does not support Intel

 

[RolloZ170]

I'm happy to try this out in windows (in fact I did the benchmarks in windows too, and got similar results). Can you point me to something?

CPU affinity
you can set the threads used in the exe.
with a batch you can direct windows to start the exe only on the selected log.processors.
please google, there a many ways with and without tools(task manager i.e.)

 

[RolloZ170]

The driver should be consistent since I was using exactly the same Ubuntu installation. I've also updated the firmware on all drives as well

i can have very different result in windows with intel or samsung NVMe driver.

 

[drdepasquale]

It could be a driver issue. NVMe SSDs should reach their maximum performance as long no PCIe switches are being used. All direct connections on the motherboard and PCIe slots are at full bandwidth.

 

[nexox]

I'd suggest trying the benchmarks without a filesystem, that's just going to add CPU load vs accessing the drive directly, though I have no idea if that particular benchmark will do such a thing and you also have to be willing to dump any data on there and reformat.

 

[autoturk]

It could be a driver issue. NVMe SSDs should reach their maximum performance as long no PCIe switches are being used. All direct connections on the motherboard and PCIe slots are at full bandwidth.

I've never had to deal w/ drivers when it comes to NVMEs -- are you suggesting something related to the chipset (I know there's not quite a chipset on the Epyc board)?

What's odd is I'm seeing the same behavior in Windows, so the driver issues would have to be across both OSs if they were the problem.

 

[rgysi]

Hi

You could try to experiment with the NUMA (NPS) settings in the Epyc motherboard BIOS to see if that helps.
Or it could just be the difference in CPU clock speed, DDR type and clock speed.

 

[drdepasquale]

I've never had to deal w/ drivers when it comes to NVMEs -- are you suggesting something related to the chipset (I know there's not quite a chipset on the Epyc board)?

What's odd is I'm seeing the same behavior in Windows, so the driver issues would have to be across both OSs if they were the problem.

I am talking about NVMe drivers in particular. NVMe drivers provided by the drive manufacturer usually perform better than the generic ones included with the operating system. The EPYC platform is not the limitation here, as there is no chipset to bottleneck the PCI Express lanes.

 

[nexox]

NVMe drivers provided by the drive manufacturer usually perform better than the generic ones included with the operating system.

I have never seen an NVMe driver provided by a manufacturer on Linux (that's kind of the point of a standardized interface,) and since the performance difference sounds consistent across both Linux and Windows, it's pretty clearly not a driver issue.

 

[autoturk]

So I'm back again -- tried reinstalling ubuntu just to triple check and same performance. I also tried a different motherboard (Asrock Romed8-2t), and also the same thing. also tried running the same benchmark on a Ryzen 7900x system I have access to and same story as the AL system -- insane performance.

I'm wondering if this has something to do with the lower single-threaded performance of the Epyc system?

EDIT: forgot to mention that I've tried binding the benchmark to the NUMA node associated with the PCIe device using `numactl`, and again no dice.

 

[RolloZ170]

I'm wondering if this has something to do with the lower single-threaded performance of the Epyc system?

match the clocks, disable intel turbo boost.