Question: Bifurcated PCIe NVMe Adapter with Raid0 and low performance

[P4in1]

Hi @ all,

I'm fairly new to Proxmox and Linux in general and i am trying to upgrade my server with some fancy hardware.
I installed the Asus Hyper with four NVMe cards on my Super Micro X12 Mainboard. I am using 4x4x4x4 bifurcation for this setup and created a RAID0 and added it as a storage for my ZFS. All NVMes were found, working as expected and then i thought i can give it a try and see how fast i get.

My assumption: 4x of those NVMes with a RAID0 setup should get me somewhere over 20GB/s for normal read and write operations.

My "Problem" : I only get 4.5 - 5 GB/s Read and Write speed which i think is underwhelming considering that a single NVMe of my setup alone should score higher than the whole RAID.

Is there some kind of setting i am missing, a switch not toggled or something else? It works, but not how i thought it should and i see videos of people getting this setup running like i wanted it, but i don't know how to "fix" this.

After reading the compatibility chart of the ASUS Hyper and seeing that it might not be compatible i bought this on from Delock, but to no avail.


My System:

Mainboard	Super Micro X12DPL-i6
CPU	Intel Xeon Silver 4314
RAM	256 GB Micron Reg ECC
GPU	3090 TI Founders Edition with GPU passthrough
ZFS Pool 1	2x 1TB NVMe Mirror direct on Board for the System
ZFS Pool 2	5x 20TB HDD RAIDZ with 1 for parity
ZFS Pool 3	1x 20TB HDD for network storage
ZFS Pool 4	4x NVMe via Bifurcated PCIe Adapter

Additional Information:

Proxmox Version: 7.3-4
The GPU is inserted into CPU1 Slot 2 and blocks Slot 3 completely (it's way bigger than i expected it to be)
The NVMe Adapter is inserted into CPU Slot 4

I would appreciate every bit of help to get this running and integrate me into the Linux space. I am still learning and trying to understand how this works in detail.
If any information would be required on how to find the solution, I will find it (but i might need some information on "how to" , because i consider myself a noob in Linux)

Thanks in advance.

Have a good Night or Day

 

[sko]

created a RAID0 and added it as a storage for my ZFS

This sounds like a very, *VERY* bad idea....
_ALWAYS_ use the disk _WITHOUT_ any intermediate raid/whatever-layers as ZFS providers!!!

Also, 4 disks without redundancy should only be used for completely worthless (test)-data, never for anything that is remotely valuable. But according to that wild mix of pool layouts I assume this is only a test system anyways...


Regarding the low throughput: Apart from the additional (sw-)raid layer that *will* affect performance and might lead to data corruption; what temperatures are those drives showing? Samsung drives are notorious for early/often and heavy thermal throttling and also running very hot even during normal/low loads.

Also, I'm not sure how/if linux handles it correctly automatically, but check the ashift size and (if present) alignment of partitions/slices. Also: in what LBA format are the namespaces formatted? For raw throughput you always want the largest LBA size and largest ZFS record size. But as raw streaming throughput is (almost) never a real-world use-case, you don't want that for a production pool because its actual performance (and space loss to padding) will be quite bad.

 

[ca3y6]

> Also, 4 disks without redundancy should only be used for completely worthless (test)-data, never for anything that is remotely valuable. But according to that wild mix of pool layouts I assume this is only a test system anyways...

well I assume the data is backed up, and if not, RAID is not a backup, doesn't protect you from a fat finger.

I don't know if RAID0 is such a bad idea for SSD. In this case they seem to be retail SSD so low endurance, no power loss protection, so ok, it's probably a bad idea. But with enterprise SSDs, the probability of failure is so much lower than for HDD that I am not sure that combining drives in RAID0 changes materially the risk of data loss*. And the most common modes of failure of SSD (electrical problems, power loss/blue screen, excess wear) would likely affect all disks in the array anyway so RAID redundancy wouldn't help.

* let's take an example, assuming zero correlation between failure. If HDD have a 5% failure rate per year, a RAID0 array of 5 disks has a 0.95^15=46% chance of survival after 3 years. With enterprise SSDs with, say, a 0.5% failure rate, survival is 93%. Not sure what is the real enterprise SSD failure rate but if not exposed to the sort of things that would kill a whole array anyway (like an electrical issue), it must be in that order of magnitude.

 

[sko]

I don't know if RAID0 is such a bad idea for SSD.

the bad idea is to put a raid0 underneath a ZFS pool. Add all 4 drives as distinct providers to a striped pool, but NEVER add another abstraction layer in front of ZFS.

 

[TRACKER]

I have setup with 4 x 970 evo 2TB NVMe in ZFS stripe config and max speed i ever got was around 9-10GB/s, usually speed is around 5-6 GB/s. Temps are fine (~36°C idle, ~47°C under heavy load).
Regarding failures - i am on the same opinion like ca3y6.
And yes - i have a backup and it is test data

 

[nabsltd]

The GPU is inserted into CPU1 Slot 2 and blocks Slot 3 completely (it's way bigger than i expected it to be)

This is very unlikely, as slot 2 is to the left of slot 3 on that motherboard, and every multi-slot GPU I have seen has all the "extra" to the left (as you look overhead, front to back of the case).

The slots are numbered 1-4 with 4 being closest to the CPUs. So, I'm guessing that the GPU is in slot 3 (2nd slot from the CPUs) and blocks slot 2.

The NVMe Adapter is inserted into CPU Slot 4

This might be correct, if it's closest to the CPUs.

According to the motherboard manual, slot 1 is connected to CPU2 while slots 2-4 are connected to CPU1.

NVMe is limited by the CPU performance, with single core speed being the biggest factor, and that CPU is designed to be wide (lots of cores), but not high (fast single core).