I'm no longer testing via FIO. But when we did, contrary to a point you made (and I do trust you) ...
When I tested a RAIDz1 of 4 NVMe, I got about 125MB/s according to ZFS I/O in performance reporting
When I tested a RAIDz2 of 8 NVMe, I got about 87 MB/s according to ZFS I/O in performance reporting
Why dont u try mirrors at all? Or at least scale slowly? and try different #jobs and qd?
I can't right now, I had to pack my server up bc I'm moving VERY soon, but what difference does that make??
Isn't that for fine tuning? If people got this performance NO ONE would use SSD drives. There's another real problem somewhere.
Are these Dell drives or non Dell drives?
The 7300 Pro are Dell ... which is why the fans don't go to 10,000 freaking-rpm with them, and why I'm limited to 4 drives until I finish moving.
Maybe proprietary BS limiting performance?
But that was why I tested them in 3 different operating systems, and get good performance for both the Dell and non-Dell drives. ;-) Remember?
Didnt u say you wanted to open a ticket?
I focused on the IDIOTIC fan issue in which with nominal CPU and exhaust temps, they punish you and cry (via their fans) bc you didn't give them extortion money ... a ransom for arbitrarily-over-priced drives with their sticker.
I also get the same-dog-crap performance using SATA SSD in RAIDz1 (4x Evo 870) via the HBA330.
Eg: Despite each Evo 870's individual performance of ~500MBs R/W ... in ZFS as RAIDz1 t
he set of 4 drives R/W performance was:
~500MB/s W (real-world, not synthetic tests of media files greater than 1GB, the same as a single drive would get)...
~680MB/s R (real-world, not synthetic tests of media files greater than 1GB, an iota more than a single drive gets)...
I didn't test SSDs in mirrors, but did a baseline of a single drive vdevs (not FIO, but actual tests) which get approx equal performance as an array.
The synthetic benchmarks & actual 'real-world' performance (copying large (1GB+) video files) in Windows and Ubuntu are the same.
The single-drive ZFS benchmarks shown are of actual testing, which is already vastly off from their performance.
That the actual single device performance and synthetic tests in Windows and Ubuntu rules out some Dell BS, right ..?
Is there anything "appropriate" or "predictable" about drives which individually get ≥ 2GB/s getting 1/3rd as an array on this (R7415) hardware..?
When an array of 8x spinning 7200-rpm drives on a T320 doesn't get a fraction of ... but multiples of the constituent devices it's comprised of..?
When Windows and Ubuntu confirm that the devices are able to perform at 2GB/s or greater ... the CPU is hardly used, with 128 PCIe 3.0 lanes..?
Also, again, it is not TNC, its your particular hw, i can get more than 750 MB with nvme easily
I think so also, especially after testing on other operating systems, which ruled several potential issues (IMO)...
And ... the first thing it illuminates is that a single device gets less than 1/3rd of what that device gets in other OS, which is wrong to me. Why?
Because ... I have seen (literally) up to 1.2GB/s in a RAIDz2 array of 8 drives which at best can get 200MB/s.
As in ... they're getting 75% of their individual performance in aggregate ... and a single drive? Has no parity to even write out.
Future testing plans:
When I get re-setup (there's no time to learn this today) I'll setup a ZFS on Ubuntu RAIDz1 array ... (which will take me time).
I might just unpack & setup Ubuntu real quick to see if I can make a mirror or RAID volume to test perf in Ubuntu...