What kind of R/W perf can I hope for with 4x NVMe drives over SFP28 using TrueNAS..?

[TrumanHW]

Did we ever talk about drive to pcie-card/multiplier to slot layout? Maybe there is your problem?
The CPU provides enough lanes, but they might not "arrive" at the slots. 24*4 lanes are 96 lanes needed for the drives alone, are those provided?

And spreading out might be a good idea, especially if that uses multiple cards...

It doesn't use any cards; they all plug into the motherboard. And I wouldn't even know how to route them to a card, as you know, those cables are lengthed to their exact connector.

With regards to lanes "arriving" ... any search terms you can suggest for me to research that ..?
Or is the fact that they aren't even going through another PCIe card somehow answering that..?

 

[Rand__]

Yes, if they all plug into the mainboard then the manual should tell you if you have enough lanes or if they are potentially over committed in Bios. How many cable do you connect and what connector do they have?

 

[pimposh]

Given the fact money spend on nice drives i won't comfort you but i had the same issue on TNS. The very same T440 in W2022/Ubuntu had nice performance but TNS chopped it halfway for PCIe3x4 drives. Did you try changing playing with I/O scheduler?

 

[TrumanHW]

I had the same issue with a T440 getting half the perf in TNS as in Server 2022 // Ubuntu with PCIe3x4 drives.
Did you try changing playing with I/O scheduler?

I re-wrote your premise to make sure I understood it ... yes?
I also can't think of the name for the schematic which shows the board's electical layout (lanes per PCIe slot, etc.)
-and-
I also searched for "schedul*" (R7415 + scheduler // schedule // etc.) and got no results relevant to this topic.

Do you have any suggestions for search terms / phrases..?
Also, what did you do to resolve your issue..?

I think one important distinction between our systems is that the T440 is intel based (Xeon Silver, gold, etc) vs Epyc.

Something's nagging at me that this might just be an Epyc issue ... though as I said:
It's NOT just in TrueNAS ... I've now replicated the issue with a RAID under Ubuntu getting the same results +100MBs R/W.

Thank you ... look forward to your reply.

 

[TrumanHW]

Yes, if they all plug into the mainboard then the manual should tell you if you have enough lanes or if they are potentially over committed in Bios. How many cable do you connect and what connector do they have?

I think this is the best indices ... https://dell.to/44w3XRU

And, this DEFINITELY looks like there are far from 96 lanes provisioned ... but rather ... only 32-Lanes.

But even still ... with just 16-lanes ... that should provide adequate lanes for 4-drives (if separated into each of those 12-bay groups) to have 100% throughput. And even though it provides 16-lanes for 12 slots (not evenly divisible ... suggesting it's flexible ... it probably isn't and provides all of 1 lane per slot? But then how does it get over 3GBs from any of the drives ... and why would they get LESS when aggregating them is still not explained).

But still, people have already told me they've gotten over 3GB/s via ZFS ...
On the next page (if you click on the link above and see the PDF) you can see that the R7525 with 2 CPUs and PCIe 4.0


BUT ... I guess I'm looking into buying a goddamned R7525 which actually has CABLES for the damned lanes that exist ... not that that explains why the F I'm getting such abysmal performance ... and for all I know..??? I could buy one and it'd be no better. :'(

Quick image of the page with the relevant image:




Here's the manual's reference: https://dell.to/44t5S9B

It's OEM configured for 24 NVMe drives (here's a picture of the configuration (highlighted) that I have):
(it didn't even support SAS nor SATA until I added the HBA330 for the first 8 slots).

 

[melindaamackenzie]

There is a very good talk by Allan Jude, that explains that if you want ZFS for NVMe, you should only be running mirrors and not any type of RAIDz
Its name is "Scaling ZFS for NVMe" and came out around 2022

Personally, i have seen abysmal performance on RAIDz but haven't really dove much further then tracing it down to latency. ZFS was tuned and developed for spinning rust, and even though it has had dramatic improvements there is still a way to go.

I can probably check our archives for the different test results i ran on a few servers when i was troubleshooting this issue.

 

[pimposh]

I re-wrote your premise to make sure I understood it ... yes?

Yes. Correct. For unknown reason both of NVMe drives fitted directly into PCIe slots are circa 1200MB/s of raw throughput vs circa 2800MB/s when ran on W2022. (no raidz of any level, just stripe across single drives)..
I did not dig into that since that T440 run with 12 spinners and it performs brilliant. With multichannel SMB i'm able to nicely saturate 1,5x 10Gbps uplink.

However this odd NVMe behaviour still exists.
I booted up from Debian12 (using 6.2 kernel branch) and it runs NVMe drives with full performance.

So i do think it's related to TNS kernel version.

Also, what did you do to resolve your issue..?

For the moment - it's not resolved.

I think one important distinction between our systems is that the T440 is intel based (Xeon Silver, gold, etc) vs Epyc.
It's NOT just in TrueNAS ... I've now replicated the issue with a RAID under Ubuntu getting the same results +100MBs R/W.

Well then our issues aren't the same.

 

[pimposh]

There is a very good talk by Allan Jude, that explains that if you want ZFS for NVMe, you should only be running mirrors and not any type of RAIDz
Its name is "Scaling ZFS for NVMe" and came out around 2022

I second that!
LINK to YT

 

[ano]

he has a 7443, he can run vdevs and high speeds, he should be able to get almost what the devices perform

fio test 1 drive, fio test multiple drives, then fio test mirrored/stripes and 4 devs z1 and 4 devs z2

 

[TrumanHW]

There is a very good talk by Allan Jude, that explains that if you want ZFS for NVMe, you should only be running mirrors and not any type of RAIDz
Its name is "Scaling ZFS for NVMe" and came out around 2022

Personally, i have seen abysmal performance on RAIDz but haven't really dove much further then tracing it down to latency. ZFS was tuned and developed for spinning rust, and even though it has had dramatic improvements there is still a way to go.

I can probably check our archives for the different test results i ran on a few servers when i was troubleshooting this issue.

HEY !!!! Thank you. To spare you having to read all the crap I posted here ... the person I bought either the unit i sent to you or the one i have (can't recall) said he got ~3GB/s (with much slower NVMe drives than those I have) in a RAIDz2 config...but even using SATA drives produced about the same results. And ... I also tested in Ubuntu (not TrueNAS, just Ubuntu) in a 3-drive NVMe array and got about the same crap.

How did YOU configure that unit ..? What kinda results did you get? Right now (bc it's electrically given access to the lanes that actually exist) ... I'm looking at getting an R7525 if not perhaps an R750 ... [IF] I'm not mistaken ... the R750 has 160 lanes (total) and also provides sufficient x4 lanes for however many NVMe drives it's configured with...the thinking being that despite Epyc actually having the lanes ... it makes no difference if there's some reason for which either Dell won't give them to us or if Epyc just isn't as good with NVMe drives. While others haven't suggested that specifically ... it may just be bc they don't know what the hell the problem with these stupid machines are.

Again ... my stupid old i7-8700k with RAID-5 and only 4x NVMe yielded ~9GB/s local performance...so I'd certainly hope that a "server" that costs multiples of these prices can at least rival some cheap consumer HW.

("melindaamackenzie" purchased an extra R7415 that I'd picked up for myself and was a total pleasure to work with).

 

[TrumanHW]

multichannel SMB i'm able to nicely saturate 1.5x 10Gbps uplink.

NICE! Were you able to use Multi-channel SMB in TrueNAS !??
What are you using to get that ..?
I'm assuming you're using 2x SFP+ cables to get that 1.5x ... yes?

I booted up from Debian 12 (using 6.2 kernel branch) and it runs NVMe drives with full performance.

Oh, remember, it does for me ALSO ... in Ubuntu ... UNTIL I configure a RAID-5 set ... at which point it craps the bed.

For the moment - it's not resolved.

:'( ... which is why I too am looking at either an R7525 (which actually gets 96 lanes for the 24 NVMe version) or an R750.

 

[TrumanHW]

he has a 7443, he can run vdevs and high speeds, he should be able to get almost what the devices perform

Can you remind the context of what that's answering..?

fio test 1 drive, fio test multiple drives, then fio test mirrored/stripes and 4 devs z1 and 4 devs z2

I HAVE I HAVE I HAVE. lol. But I certainly don't blame you for not reading all this crap.
Still ... I TESTED ZFS with 1x NVMe ...(with FIO and real-world).

Testing 1 drive, 3 drives in Rz1 and 8 in Rz2 told you "nothing" apparently ... but now that I added one more test with mirrors that suddenly does tell you something..? What?

They ALL yield the same performance of ~600MBs Write, 700MB/s Read.

The poor performance I get with 1 drive only gets WORSE "performance per drive" as drives are added ...
But it really will tell you something ONLY after I tested mirrors..? And before then, scaling perf. down tells you NOTHING !????

 

[ano]

did you fio test the device as well? /dev/nvme0n1? etc ? (without filesystem)

 

[melindaamackenzie]

Heres the FIO tests on TrueNas Scale on the Dell PowerEdge 7415 I purchased from you.
same memory 128GB
Number of disks 10
disk size 30.72TB

 

[melindaamackenzie]

Heres the FIO tests on Ubuntu Server 22.04.02 on the Dell PowerEdge 7415 I purchased from you.
MDADM --raid10
same memory 128GB
Number of disks 10
disk size 30.72TB