2nd zpool or add VDEV to existing zpool?

[eldakka]

Hi, I'd like to ask for some advice, as I'm mulling amongst 2 different options.

I have a 7-disk raidz2 pool, using 8TB 7200RPM HDD that is at 97% capacity (I know I know, should keep below 80% for performance, but it's mostly video, so don't really need a lot of raw performance, price:capacity ratio is a bigger factor than raw performance).

I have 5 unused 8TB 7200RPM HDD available.

What would be the better way to go, create a 2nd 5-disk raidz2 pool, and migrate some of the (still growing) datasets to it from the first pool, or buy 2 more HDD so I'd have 7 disks to add to the existing pool as a second 7-disk raidz2 VDEV? I understand that the best practice with multiple VDEVs is for them to be identical in terms of number of disks, vdev type, capacity, and performance, hence buying 2 more disks to match those characteristics with the existing VDEV.

The latter option would save a lot of work in doing dataset migrations between pools, and easier to do other fancy stuff if I decide to do it later like adding a metadata special vdev or other similar configurations since I'd only have one pool for that to apply to, rather than multiple pools each requiring their own independent configurations and devices. But it has obvious downside of immediately costing money I otherwise wouldn't spend in buying an extra 2 HDDs (although the money isn't a problem if that's the better way to go).

Are there any other options I'm missing? Thoughts on which would be the better option?

_____________________________________________________________________
Supermicro X11SSL-CF
Intel E3-1240L v5
32GB ECC RAM
Intel RES3TV360 expander
Linux 5.15.52-gentoo

 

[aero]

This may or may not matter to you, but it irks me to no end....if you add another vdev to your pool all your writes will go to the new vdev, they will not be balanced.

 

[i386]

Option 3: backup your stuff, delete exisiting vdevs and create a single z2 (or z3) vdev with all the 8tb hdds
(I know that you probably don't have the extra storage for an entire backup)

To be honest I think all options suck with zfs (or storage spaces) when it comes to expansion. Losing 4 of 12 hdds to parity would be too inefficient for me...
Also buying "new" 8tb hdds in 2022 (almost 2023) is kinda weird: even in europe the sweet spot is currently 16/18tb hdds.

Maybe option 4? buy new larger hdds, move all the stuff to this storage and repurpose the "old" 8tb hdds for a backup server or something else?

 

[aero]

When the heck is the version of ZFS on linux coming out that allows 1. expansion of raidz vdevs, 2. automatic rebalancing of data

 

[i386]

When the heck is the version of ZFS on linux coming out that allows 1. expansion of raidz vdevs, 2. automatic rebalancing of data

The pullrequest is not approved/reviewed yet and there are failing builds and tests: RAIDZ Expansion feature by ahrens · Pull Request #12225 · openzfs/zfs
Also when you read all the comments of that pullrequest you will see that this feature won't work like in "traditional raid" (due to zfs design) and will be pretty inefficient when adding single devices...

 

[eldakka]

This may or may not matter to you, but it irks me to no end....if you add another vdev to your pool all your writes will go to the new vdev, they will not be balanced.

Ah yes, thanks for the reminder. I'd probably want to rewrite all the data, copy it all to a temp file/directory (e.g. file.new), delete the original, then mv it back. Painful but doable.

 

[eldakka]

Some good suggestions, all of which I'm mulling over to see if they fit within or modify my resources and plans.

Option 3: backup your stuff, delete exisiting vdevs and create a single z2 (or z3) vdev with all the 8tb hdds
(I know that you probably don't have the extra storage for an entire backup)

I do have an LTO-5 drive and tapes, but that sounds rather tedious since it's a bare drive and not an autoloader/robot, so that's a lot of tape swapping

Although really I should do a full backup anyway since I have the drive and tapes. I'm just lazy ...

To be honest I think all options suck with zfs (or storage spaces) when it comes to expansion. Losing 4 of 12 hdds to parity would be too inefficient for me...
Also buying "new" 8tb hdds in 2022 (almost 2023) is kinda weird: even in europe the sweet spot is currently 16/18tb hdds.

Maybe option 4? buy new larger hdds, move all the stuff to this storage and repurpose the "old" 8tb hdds for a backup server or something else?

I hear you with ZFS issues. It's pretty cool, but not perfect ...

I tend to find that from an up-front-cash perspective for personal/homelab use, around AUD$250 per drive is the sweetspot. So when 1TB were around that price, I got them, then 1.5TB -> 3TB -> 8TB all at around AUD$250. But the bigger HDDs seem to be stubbornly remaining at or above - well above for 16TB+ - $300 for a few years now, with 8TBs at around $250 for 2 or 3 years now, prices haven't come down like they have historically. It's been rather a stagnation. Like with GPUs I guess, where NVIDIA/AMD seem to be increasing the price points rather than offering more for the same $$, but that's life/inflation/economics I guess.

I've had the 8TB's for something like 2-3 years now. Wow, 4 for the older ones, doesn't seem that long. If I was starting from scratch today, I'd certainly be looking at - probably - 14TB HDDs for that amount of storage, as they are the best $/GB at the moment in Australia according to au.pcpartpicker.com, 0.026c/GB. But at ~AUD$370 each, for the 7 I'd need for raidz2 thats AUD$2500 up front. The disks you have are better than the ones you have to fork out over $2k for

One of the reasons I was thinking of 2 VDEV's instead of 1 big 10 or 12 disk one is so that I can eventually get the bigger HDDs and 'upgrade' 1 VDEV at a time to bigger HDD's when - hopefully - they come down a bit more and I can justify the expense to myself. That was my original plan with my original 5x8TB pool from 4 years ago, but as I said earlier, prices seem to have stagnated for years in the $250ish sweetspot, which I guess is no longer the sweetspot, looks like that sweetspot may be moving to AUD$350, damn inflation!

However using the old 8TBs as backup media sounds tempting with moving to newer bigger HDDs instead for the 'live' array despite the upfront cost - more appetizing than using LTO-5 1.5TB tapes for backing up that much data.