RAID-5 plus NVME filesystem terrible performance on Linux

[LikesChoco]

I don't believe that.

Yeah, these tests should see massive write combining for large transfers. I mean, even if they were synchronous writes when I give it the easiest possible target, a 1MiB write it doesn't appear to help. At the filesystem layer it should be allocated and then written in a large transfer down to the md layer. Then at the MD layer they should be broken up into stripes and then write combined before writing to the devices. A bunch of splitting and coalescing should be happening for these transfers and I don't see any evidence of that.

Based on single device performance I believe I should conservatively be seeing >2GB/sec and possibly >3GB/sec writes for favorable workloads like 1MB writes.

 

[LikesChoco]

Different write types and sizes directly on the RAID-5 device. No filesystem was involved.
Queue depth 16.
I/O Type is: (b)uffered, (d)irect, (f)sync
fsync is performed at the end of the test.

Transfer Size	I/O Type	MB/sec	IOPS
4	b	564	141004
16	b	656	40994
64	b	680	10618
256	b	504	1970
1024	b	711	694
4	d	336	83879
16	d	676	42274
64	d	1182	18465
256	d	2530	9882
1024	d	3562	3479
4	f	3	637
16	f	10	616
64	f	41	637
256	f	121	471
1024	f	317	309

What strikes me is how bad buffered I/O is. Here's a config file. Did I do something wrong?

[global]
name=write_test
size=40G
bs=16K
rw=write
direct=0
numjobs=1
iodepth=16
ioengine=libaio
end_fsync=1
group_reporting=1

[job1]

 

[unwind-protect]

That is indeed odd.

I would repeat the tests inside a large file inside the filesystem.