I'm well aware of that, no worries.
M2 PCIe is PCIe 4x4 per slot, so 8GB/s per slots, which is more than enough. With 3 slots it's 24GB/s (in theory).
If you add an additional NVMe plugged in a PCIe 4 16x slot, it's now 4 NVMe, each working at PCIe x4, so 4x8 = 32GB/s max. This NVMe is announced at 7GB/s, but real test gives me a steady 6GB/s per NVMe, as stated in my initial post. With 2 NVMe drives in strip or mirror, I got 12GB/s.
If you read my first post, you will see that I can create 2 pools of 2 NVMe each (one with 2 on M2 slots, and one with 1 M2 slots and one on a PCIe card), and each pool got 12GB/s. So it's not a PCIe link degradation problem.
Welcome to "Combined PCIe lanes" you can make the x16 go slower but you can never make an x4 go faster. Let me simplify it...You're on the wrong (PCIe) BUS...
TLDR solutions Not enough PCIe lanes on the chipset for the amount of storage drives attached directly to the motherboard. Move all pool drives to PCIe x16 card of some type.
The TRX40 you have only has 16 PCIe lanes on the chipset and that's what the on board m.2 slots use exclusively.
The M.2 slots are all capable of PCIe 4.0x4 but only under specific conditions.
That motherboard only has 16 total PCIe lanes on the board. 8 reserved for the chipset and 8 lanes dedicated to the m.2 and SATA III ports.
You get 4 lanes on the bottom slot only if there's nothing detected in POST on the SATA controller.
And/Or
You get 4 lanes in either the top or the middle m.2 slot when only one or the other is populated, if both are populated then its 2 lanes each.
The long version...links sources ect at bottom. If you think I'm full of crap watch the LTT video.
The M.2 slots on your MB only have 8 lanes between themselves and the SATA III controller.
Allocation as follows..
'2 x Hyper M.2 Sockets (M2_1 and M2_2), support M Key type 2260/2280 M.2 PCI Express module up to Gen4x4 (64 Gb/s)*'
this is gigabits not bytes, someone will inevitably bring that up. 8GB/s max per m.2 slot but not in every slot all at once.
These two slots get 2 lanes each, it may be 4 lanes if only one is populated. Most likely un-effected by adding other devices, though your mileage may vary.
'1 x Hyper M.2 Socket (M2_3), supports M Key type 2230/2242/2260/2280/22110 M.2 SATA3 6.0 Gb/s module and M.2 PCI Express module up to Gen4x4 (64 Gb/s)'
This slot, oddly enough is the bottom one and as with many many others the bottom generally is the "lowest priority" M.2 slot, is sharing its PCIe 4.0 x4 with the MB SATA III ports. Once the bios detects something on the SATA bus its going to split the bottom slot into 2 @ PCIe 4.0x2, one for the SATA controller and one for the NVME drive.
All 3 m.2 slots will support SATA m.2 drives but only the the top two will boot from an NVME m.2.
Not 100% on why or how but it feels like splitting the drives between a PCIe card and the m.2 slots on the motherboard, from what I can gather causes the PCIe bus to pseudo hand that slot off to the chipset for management or maybe it just matches the chipset Gen and link width I haven't been able to get a straight answer on this one. This cannot work the other way around thought do the the physical limitations of the M.2 key(s) standard only being 4 lane pinout.
The only ways I know to pass the 12GB/s is to put all of the drives in the pool on a single PCIe 4.0X16 slot and boot from the motherboard m.2 slot with an NVME drive not in the pool. You can still boot from an M.2 NVME but don't pool it with the other drives. This puts the entire pool on the CPUs PCIe lanes.
The other option...boot from a PCIE card and run ONLY the M2_1 and M2_3 slots. Leave M2_2 empty as well as leaving the motherboard SATA empty. Then you get 2 x M.2 slots @PCIe 4.0x4 totaling 16GB/s in theory. Populating the top two slots is how you end up back at 8GB/s (6GB/s in your test from the looks of it)
I would use the slot 1 PCIe 4.0x16 for the NVME m.2s in a raid card(unless GPU is needed at x16 link) and slot 3 for the next most important item. Leave the board m.2 spots for a boot drive and maybe a Cache if it'll let you. I know TrueNAS doesn't like mixing CPU and Chipset lanes in the same pools. It looks fine, doesn't throw errors but it straight up ignored the 2tb NVME cache drive on the motherboard and wrote directly to the HHDs in the pool when I tried a few weeks back.
Additionally your 4 @ PCIe 4.0x16 slots are technically two 4.0x16s and two 4.0x8s due to slot 2 and slot 4 splitting an x16 set when all 4 slots are populated. So like, in theory you could run 4xGPUs but two would be cut down to 8 lanes each while the other two were free to run at x16.
'- 4 x PCI Express 4.0 x16 Slots (PCIE1/PCIE2/PCIE3/PCIE4:
single at x16 (PCIE1);
dual at x16 (PCIE1) / x16 (PCIE3);
triple at x16 (PCIE1) / x16 (PCIE3) / x8 (PCIE4);
quad at x16 (PCIE1) / x8 (PCIE2) / x16 (PCIE3) / x8 (PCIE4))* '
Sources & Tools used to reach these conclusions.
PCIe Gen 4 is the latest and fastest PCIe standard available on motherboards and compatible CPUs. PCIe Gen 4 cards can accommodate the full bandwidth that is possible with NVMe SSDs technology.
www.kingston.com
Supports 3rd Gen AMD Ryzen™ Threadripper™ Processors; 8 Power Phase Design, 90A Dr. MOS & Power Choke; XXL Aluminum Alloy Heatsink & Heatpipe Design; Supports DDR4 4666+(OC); 4 PCIe 4.0 x16; NVIDIA NVLINK™, 4-Way SLI™, AMD 4-Way CrossFireX™; 8 SATA3, 2 Hyper M.2 (PCIe Gen4 x4), 1 Hyper M.2 (PCIe...
www.asrock.com
LinusTechTips has a great video about chipset vs CPU lanes here >>
Me who added an m.2 drive to a system with 1 m.2 and 8 SATA SSDs plugged directly into the motherboard and couldn't figure out why the bios would only see the M.2 drive in the top m.2 slot.
