ZFS and # of drives question

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,292
1,756
113
CA
Changing my home storage and I have some spare drives I'm wondering if I'm not better off using them.
Wondering too if anything has changed with how I should be doing things.

System: DIY build with--> TrueNAS, Intel 9100F, 16 or 32GB, mirror SSD for boot\os.

ZFS Config: RaidZ3

Use case:
  • family pictures from phones, DSLR & mirrorless cameras, etc.
  • raw\original videos to be used in premiere
  • final videos created
  • desktop documents \ files backup

I'm currently using 8x10TB HGST enterprise drives but am wondering if I should just use the other 2 or 3 I have for 10x10TB or 11x10TB and how this would work with RZ3, and if it's a concern for the overall setup in regard to: 1) size of rz3 and 2) number of drives utilized for rz3. As you can tell from my use-case list above sequential performance is what's important to me. I can of course get 1 more drive if I need that number of drives to make sense but feel like I'm getting up there in size for a single pool.


Thoughts?
 

BoredSysadmin

Not affiliated with Maxell
Mar 2, 2019
786
291
63
do you have any idea what sort of bandwidth you need working with premier raw files? My gut feeling is 10-11 drivers in a single Z3 raid ground won't give you the performance you want. did you consider Two striped Z2 vdevs of 6 drives each? You won't be sacrificing much in terms of reliability. You'd probably also want a fast optane slog drive.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,292
1,756
113
CA
This is for storage only, the work drives will be on the video system and be a combo of NVME and HDD.
So just moving files to\from the systems not working on while stored.
Should have mentioned that above, good catch :)
 

PithyChats

Active Member
Feb 3, 2015
166
82
28
Changing my home storage and I have some spare drives I'm wondering if I'm not better off using them.
Wondering too if anything has changed with how I should be doing things.

System: DIY build with--> TrueNAS, Intel 9100F, 16 or 32GB, mirror SSD for boot\os.

ZFS Config: RaidZ3

Use case:
  • family pictures from phones, DSLR & mirrorless cameras, etc.
  • raw\original videos to be used in premiere
  • final videos created
  • desktop documents \ files backup

I'm currently using 8x10TB HGST enterprise drives but am wondering if I should just use the other 2 or 3 I have for 10x10TB or 11x10TB and how this would work with RZ3, and if it's a concern for the overall setup in regard to: 1) size of rz3 and 2) number of drives utilized for rz3. As you can tell from my use-case list above sequential performance is what's important to me. I can of course get 1 more drive if I need that number of drives to make sense but feel like I'm getting up there in size for a single pool.


Thoughts?
I'd say this is the definite answer wrt storage efficiency.

Note that setting a small recordsize with 4KB sector devices results in universally poor space efficiency -- RAIDZ-p is no better than p-way mirrors for recordsize=4K or 8K. The strongest valid recommendation based on exact fitting of blocks into stripes is the following: If you are using RAID-Z with 512-byte sector devices with recordsize=4K or 8K and compression=off (but you probably want compression=lz4): use at least 5 disks with RAIDZ1; use at least 6 disks with RAIDZ2; and use at least 11 disks with RAIDZ3.

from


(Matt is an OpenZFS dev)

For some real life data on sequential performance see:


I'd probably go 11 drives in your case - performance should be plenty for your use case (I'd guess you can pretty much saturate 10G on reads).
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,292
1,756
113
CA
Great links, here I thought I'd seen all the ZFS articles but those were new :)

Looks like I found another drive, and have 12x10TB I can use for a RZ3 pool.

using the 12x drives for a RZ3 pool with 9 sector block is most efficient?

 

PithyChats

Active Member
Feb 3, 2015
166
82
28
Great links, here I thought I'd seen all the ZFS articles but those were new :)

Looks like I found another drive, and have 12x10TB I can use for a RZ3 pool.

using the 12x drives for a RZ3 pool with 9 sector block is most efficient?


9 sector multiple block size is the most efficient, yes - but I wouldn't stress too hard about it. Remember that you can still have fragmentation based on your block size and lose efficiency there. As a rule of thumb, I'd pick a block size that is a power of 2 (at least 8) and aligns most closely with your use case. Compared to 8, 9 is better, but say you need larger blocks, I would not use 9 over say 64 just to have a slightly higher space efficiency. Note that drive format matters here. A 9 x 4KiB is 36KiB while 9 x 0.5KiB is only 4.5KiB. If you are storing mostly media, and in particular want sequential performance, you likely want much larger blocks. Recall that smaller block size means more IOPs for accessing large files - thus for strong sequential performance on large files, you generally want a larger block size. Block sizes of 128KiB is often used as a rule of thumb for general workloads, but you may want even larger - 256 * 4KiB is 1MB block size, which may work well if your data set is mostly media. Even at 128KiB, you are still looking at 32 * 4KiB. Both of these give you essentially the same storage efficiency with 26% and 27% respectively, compared to 25% for 9.

Finally if most of your small files are stored in backups rather than directly, note most backup programs use larger block size. Veeam for example uses 1MB blocks, though these end up smaller after compression. Veeam recommends 128KB or 256KB blocks. See:


Ultimately, these kinds of things greatly matter in large scale deployments and data centers, but for home use, you likely won't actually notice a difference unless you choose very esoteric values. Nevertheless, it is worth thinking about a bit, especially about which data-sets are most likely to be prioritized. For example a small text files is always going to be 1 iop, pretty much regardless of block size. But a large video file could see some variation in performance.

Finally note that there is also record size. This discussion may prove helpful:

https://www.reddit.com/r/zfs/comments/8l20f5
as may this:


and this


I'd be curious on what you settle on, and what performance you see.
 
Last edited: