SOLVED: Anyway to improve the performance of new TrueNAS rig (EPYC 7402P)

[traderjay]

THANKS all for the suggestions. I solved my issue by creating a 3 x 2(disk) mirror and now I am able to edit and scrub the footage smoothly! Performance went up by ALOT!

So just recently built a new TrueNAS core rig with the following spec:

CPU - EPYC 7402P – 24 core

RAM - 256GB DDR4 ECC RAM

Motherboard - ASROCK EPYCD8

NIC: Broadcom 10g RJ45

Main Array:

6X10TB WD Red NAS Pro in Raid Z2


The drives are mounted in windows 11pro as SMB

I am trying to edit my 8K RAW footage directly from the server, and the video bitrate is 1700 Mbps (250MB/s roughly). I did some file copy test and was able to achieve a minimum of 400MB/s write and 10000 MB/s read speed on a large 75GB video file. Yet the playback is choppy in Davinci Resolve… Edit works file and smooth when using local NVME drives.

Is my write speed reasonable for a server of this spec? Anything else I can optimize?

 

[i386]

If the mentioned read and write speeds are correct then it's not the storage.

Is the client/editing host using a gpu/video acceleration?

 

[Blue4130]

So just recently built a new TrueNAS core rig with the following spec:

CPU - EPYC 7402P – 24 core

RAM - 256GB DDR4 ECC RAM

Motherboard - ASROCK EPYCD8

NIC: Broadcom 10g RJ45

Main Array:

6X10TB WD Red NAS Pro in Raid Z2


The drives are mounted in windows 11pro as SMB

I am trying to edit my 8K RAW footage directly from the server, and the video bitrate is 1700 Mbps (250MB/s roughly). I did some file copy test and was able to achieve a minimum of 400MB/s write and 10000 MB/s read speed on a large 75GB video file. Yet the playback is choppy in Davinci Resolve… Edit works file and smooth when using local NVME drives.

Is my write speed reasonable for a server of this spec? Anything else I can optimize?

I'd be looking into if it's a latency issue. 6 drives in raid over ethernet are bound to have more latency than a local nvme. Does it happen more when you are scrubbing and jumping around the timeline?

 

[sko]

what you need for that use case is more IOPS - so spinning disks are already sub-optimal (read: bad), and RAIDZ doesn't give any improvements. Add a horrible protocol like SMB to the mix and real-life performance goes down the drain...
ZFS *needs* to spread the load over multiple vdevs - so especially with spinning rust where a decent IOPS performance is still required use mirrors! Those are also much more flexible, as any RAIDZ configuration is carved in stone and you have to follow through with that or destroy and recreate the pool (or some heavy restorations with temporary mixed configurations...)

To remedy the situation somewhat, you could add a mirror of special vdevs, ideally on NVMe or at least on some fast (=SAS) SSDs. This will at least shift the random-IO-heavy ZFS metadata operations and housekeeping from your spinning disks and lets them do more sequential work.

But even if that pool would be local, spinning disks are multiple orders of magnitudes slower than NVMe, so of course you won't get anywhere near their performance, regardless how many HDDs you throw at a ZFS pool.

I'd suggest using the HDD pool purely as a data grave and add a dedicated, flash based pool for data that is being worked on.

 

[ano]

should be faster

can you run a fio locally on the zfs ?
and iperf

 

[gea]

If you edit videos, you will concurrent read/write multiple files where iops become relevant. This is much slower than a single large sequential file read/write. To improve iops you can use 3 mirrors instead a single Z2 vdev. This will reduce usable capacity from 40TB to 30TB but increase write iops by a factor 3 and read iops by a factor 6. You should also use a larger recsize ex 1M instead the dafault 128k.

Another tuning option is a special vdev mirror (nvme/ssd) that can hold small io like metadata or a whole filesystem with a recsize setting smaller a threshold. If you buy new disks, prefer 2x12G multipath SAS over 6G Sata.

 

[ericloewe]

Another tuning option is a special vdev mirror (nvme/ssd) that can hold small io like metadata or a whole filesystem with a recsize setting smaller a threshold.

That's not likely to help in this case. Surely any bits of metadata would get handled easily by the ARC.

f you buy new disks, prefer 2x12G multipath SAS over 6G Sata.

TrueNAS doesn't even support multipath anymore, it was always more trouble than it's worth. I also question the wider idea of using SAS disks: For spinning rust, it won't make a difference. For SSD, NVMe is better and cheaper in most situations and certainly on an Epyc.

 

[jei]

I'm not yet doing 8K but for photography/4K for this reason I went from HDD pool to NVME, no regrets. Even just browsing files and reading their metadata was slow as sh*t with HDD RAIDZx.

 

[unwind-protect]

Try mounting as iSCSI, not SMB.

Always seperate out reading and writing if they happen at the same time. With spinning disks they need to go to separate arrays.

 

[ericloewe]

Try mounting as iSCSI, not SMB.

Yeah, no, that's a genuinely terrible idea. How would you expect that to improve anything? This is about files, large files at that, why would you add the performance penalty of block storage to an existing performance problem?

Always seperate out reading and writing if they happen at the same time. With spinning disks they need to go to separate arrays.

That's nice in fantasy land. In the real world, people need to read what they wrote.

 

[traderjay]

Thanks all for the replies. The thing is video editing doesn't need too much random IOPs since it's just a large chunk of sequential file? Guess I am SOL lol... The thing is my read speed is great and I figured when editing on davinci resolve that matters more...

 

[gea]

That's not likely to help in this case. Surely any bits of metadata would get handled easily by the ARC.

TrueNAS doesn't even support multipath anymore, it was always more trouble than it's worth. I also question the wider idea of using SAS disks: For spinning rust, it won't make a difference. For SSD, NVMe is better and cheaper in most situations and certainly on an Epyc.

Arc is readcache only, special vdev improves small io on read and write,
but you get most of a special vdev if you force a whole filesystem even with larger files onto.
For such a filesystem performance is identical to SSD/nvme pools. Other filesystems that are not as performance sensitiv land on slower disks.

To test if this helps you can just create a testpool on an SSD/NVMe. If results are much better than with the hd pool, you can get the same with a special vdev for one or more filesystems.

SAS multipath can double performance on concurrent io but even without SAS is more robust, with 12G twice as fast as Sata
and offers full duplex (concurrent read/write) while Sata is half duplex. Especially with SSD you can achieve a performance near to NVMe while SAS advantages persist (up to hundreds of disks, hotplug, costs etc)

 

[amalurk]

Video card in there you didn't list for hardware acceleration of 8k playback?

 

[unwind-protect]

Yeah, no, that's a genuinely terrible idea. How would you expect that to improve anything? This is about files, large files at that, why would you add the performance penalty of block storage to an existing performance problem?

It moves the filesystem VM cache from the far side of the network to the near side. If the client machine has lots of RAM that can make a big difference.

That's nice in fantasy land. In the real world, people need to read what they wrote.

Well, writing and then reading is fine. But doing both at the same time is not, especially not on spinning disks.

 

[traderjay]

Video card in there you didn't list for hardware acceleration of 8k playback?

My editing PC has 4090, 64GB DDR5 ram and 7950X3D so that rig is not holding the performance back.

 

[ericloewe]

Arc is readcache only, special vdev improves small io on read and write,

How much small I/O are you seriously doing in this situation? Not much, I'd wager. How many sync writes? Between zero and none, so it all gets aggregated in RAM and flushed to disk in one go every 8 seconds or so. Hardly a demanding workload.

but you get most of a special vdev if you force a whole filesystem even with larger files onto.
For such a filesystem performance is identical to SSD/nvme pools. Other filesystems that are not as performance sensitiv land on slower disks.

At the point where you have a separate fast dataset, you might as well have a separate fast pool, since you get none of the advantages of pooled storage.

SAS multipath can double performance on concurrent io but even without SAS is more robust, with 12G twice as fast as Sata
and offers full duplex (concurrent read/write) while Sata is half duplex.

I'm all for SAS controllers, but not SAS disks.

Especially with SSD you can achieve a performance near to NVMe while SAS advantages persist (up to hundreds of disks, hotplug, costs etc)

"Near to" is doing a lot of work in that sentence, especially if you're latency-bound. Cost-effectiveness is also not something typically associated with SAS anything, Broadcom and Microchip made sure of that. Hundreds of disks and hot-plugging are indeed more mature on SAS, but that's far into the deep end.

It moves the filesystem VM cache from the far side of the network to the near side. If the client machine has lots of RAM that can make a big difference.

I can imagine such an edge case, but it's a very contrived scenario and certainly not applicable to video editing (large files, lots of bandwidth, low latency for seeking).

 

[unwind-protect]

I can imagine such an edge case, but it's a very contrived scenario and certainly not applicable to video editing (large files, lots of bandwidth, low latency for seeking).

I'm not sure it will be worth it with "only" 64 GB on the client. But it is no cost to try and maybe fun as a learning experience.

Latency will certainly be lower for larger parts of the files in iSCSI. Again, assuming enough RAM.

 

[BoredSysadmin]

I agree with the posters above, especially the idea that a pool of spinning rust is a wrong choice for 8k video NLE.
Your editing workflow should be similar to the guy in this video. Davinci itself is installed on local nvme. Your projects and rendering on larger (and cheaper) SATA SSD and long-term storage are acceptable on your truenas with hard drives.

 

[unwind-protect]

What the OP wants is not unreasonable at all. He/she gets enough performance out of the SMB drive to cover 4 times the required bitrate of the video. Yet it is choppy for what is presented as a simple linear read. The magnetic drives obviously have enough linear read performance to theoretically make this work - if it wasn't for the networked drive mechanism, which is screwing things up.

Trying iSCSI is free and might shake things up quite a bit.

 

[jei]

Not enough info about the resolve project. Propably simple one asset 1x playback on timeline shouldn't be choppy if the pool is not heavily fragmented.