raidz1 hotspare question/enlightenment

[jcizzo]

Hey folks! i'm reaching out because i've heard 2 different statements to the same question. This pertains to zfs not being like hardware raid (or any other raid for that matter).

i'm building a truenas core 13.x nas (latest stable) and I had an idea: i have 5 x4TB spinners in one of those 5 bay raid cages that pops in the 3x5.25 drive bays.
i thought i could also add a single 4TB SSD as a hot spare because if it's just sitting there waiting to go into service when a spinner dies, it's not using any power and not wearing out like a spinner does.
i thought the way it works/worked was, upon the death of one of the spinners in the raidz1 array, the SSD (of equal size, of course) would temporarily take over. zfs would use the parity yadda yadda and rebuild the dead drive on the SSD. then, upon replacement of the dead spinner with another spinner (same model as the other spinnners), the data would be automatically moved from the SSD to the new drive, the SSD would go back into an idle state and await to be reused when the next drive dies..

some of you are probably asking why i would do it like this? well, if i'm away from my nas and i can't be there to immediately install a replacement drive, the SSD would help it along (like a donut spare tire) until i get back to replace the drive. the nice thing about this would be, again, no wear while the SSD is at idle and a theoretically much quicker rebuild time.

I can see how this wouldn't work with hardware raid or any other software raid, but considering ZFS seems to have a lot of intelligence built into it, i thought maybe it's possible.

I've been told "yes, it can do that" by someone on a truenas forum. i've been told "no, it can't do that" by someone who has a TON of experience, just none with ZFS.

Thanks for your patience!

 

[oneplane]

And old but relevant reference: ZFS hot spares versus more parity

Depending on the CPU, it's not raidz1 you want, but 2x mirroring + hotspare with this specific disk count. While you can definitely just do what you want (only the hot spare coming in to save the day is automatic, everything else requires you to tell ZFS what it is you want), it's not exactly safe for your data, and performance-wise you may be mostly CPU and RAM bound anyway (need specs for that). If not, go for a 5-disk raidz2.

 

[reasonsandreasons]

I'll second the mirror recommendation (for what my homelabber opinion is worth). People really underrate how stressful RAIDz resilvers are on the pool, and the slightly higher theoretical redundancy of RAIDz2 often doesn't beat the practical benefits of much less stressful rebuilds. Plus performance is a lot better. In this application you'd lose some storage space in exchange for more safety, but you're not exactly bumping up against the limits of what your enclosure can support; 4TB HDDs are pretty low-end these days.

One meta-point I'd feel remiss not bringing up here is that flash is cheap right now and the pool you're building isn't that big. It's worth asking if you could just go all-flash from the beginning if you want the benefits of it in your hot spare.

 

[oneplane]

One could argue 2x6TB flash in a ZFS mirror would be fine But in some aspects I am in the same boat as I also have a ton of disks varying from 1TB to 6TB and I just can't bring myself to e-waste them and just get 6x22TB drives...

 

[Rand__]

He's looking for 5x4 TB in Z1, so 16GB usable, + a 2,5" 4TB hotspare ... - so 6TB mirror won't do it.

Else I dont think it would use no power if its a hot spare, its connected to power so it will run (even if in deep sleep), but maybe just a little.
If you do that then make sure that the SSD is actually able to work as spare, ie make sure it has enough blocks/size - 4TB may not be 4TB as ppl with different brands of xTB drives have learned the hard way.

 

[oneplane]

He's looking for 5x4 TB in Z1, so 16GB usable, + a 2,5" 4TB hotspare ... - so 6TB mirror won't do it.

Else I dont think it would use no power if its a hot spare, its connected to power so it will run (even if in deep sleep), but maybe just a little.
If you do that then make sure that the SSD is actually able to work as spare, ie make sure it has enough blocks/size - 4TB may not be 4TB as ppl with different brands of xTB drives have learned the hard way.

Ah you are right. I did see his disk size and count but assumed he wasn't looking for a specific pool capacity, but if he is that wouldn't work.

 

[jcizzo]

sorry for not filling you all in on the specs of my system..
(thank you all for responding so quickly, btw )

it's based on an i3-7100T, on a supermicro X11ssh-f motherboard with 32Gs of ecc ram and an LSI 9200 based hba in IT mode (of course).

its purpose is to be a relatively simple, bare metal home nas. i won't be running (at least i don't think i'll be running.. maybe urbackup or something.. but i digress) packages.. it'll have 4 x ssds (2 for the OS, 2 larger ones for important files, mirrored). the 5 spinners will be for storing media only .. movies, tv shows.. i run plex, but the plex server is on a completely different system because that does a whole mess of other things that actually DO require power..

the way it'll work is, a share will be made, drive mapped to the media server and a dedicated (via dac) cable run from the nas to the mediaserver. media streams from the nas to the mediaserver, and from there out to whoever wants it.

the 5 spinners? meh... i'm definitely going to be backing those up on a separate drive and if i lose the pool i can just rebuild it and copy the media back from an external drive.

cheap is relative.. yes, SSDs/flash have come down a ton in price, but i''m closing in on 10TB's of movies and there's more to come.. PLUS these drives that i have were free, so... several large SSD's is prohibitively expensive compared to the same number of free spinners.

i'd prefer to use them until i can afford to go all SSD

 

[jcizzo]

I'll second the mirror recommendation (for what my homelabber opinion is worth). People really underrate how stressful RAIDz resilvers are on the pool, and the slightly higher theoretical redundancy of RAIDz2 often doesn't beat the practical benefits of much less stressful rebuilds. Plus performance is a lot better. In this application you'd lose some storage space in exchange for more safety, but you're not exactly bumping up against the limits of what your enclosure can support; 4TB HDDs are pretty low-end these days.

One meta-point I'd feel remiss not bringing up here is that flash is cheap right now and the pool you're building isn't that big. It's worth asking if you could just go all-flash from the beginning if you want the benefits of it in your hot spare.

I've considered doing mirrors for the spinners as well.. (buying 2 x10TB and then as funds permit, buying another matching set and adding them so i'd have 2 mirrored vdevs)..

originally i was going to just do raidz1 with the 5 drives and add the aforementioned SSD as a hotspare.. however, i read an article the other night that brought to my attention precisely your point: it detailed the stress put on drives during a resilver, which i wasn't aware of... that coupled with being told i "can't have a spinner based raid5 with an SSD based hotspare" and that made a mess of my plans

but, as i mentioned, the person who told me i can't do it that way, while incredibly knowledgeable has NO experience with zfs, so i brought my dilemma to you folks!

 

[jcizzo]

And old but relevant reference: ZFS hot spares versus more parity

Depending on the CPU, it's not raidz1 you want, but 2x mirroring + hotspare with this specific disk count. While you can definitely just do what you want (only the hot spare coming in to save the day is automatic, everything else requires you to tell ZFS what it is you want), it's not exactly safe for your data, and performance-wise you may be mostly CPU and RAM bound anyway (need specs for that). If not, go for a 5-disk raidz2.

I've read that article you've sent me before.. i guess i could do the 5 disk raidz2, but again, i'm starting to run outta space.. plus i figured raidz1 across 5 drives would be more performant.. i don't recall ever seeing, but with zfs, there isn't a way to specify a parity drive, correct? it just stripes parity across all drives (if i understood it correctly)..

the other part that i forgot to mention is that the whole reason why i'd like to use this drive bay that i have is because of its built-in fans.. gotta keep those spinners at a proper temp. PLUS it makes them safely hot-swappable..

what could be cool (if it were possible) is the 5 spinners for storage and 1 (or two mirrored) ssd's for parity.. but i don't believe any of this works in that manner.. if it did, i'd assume that would alleviate a lot of the stress during a resilver.. correct?

i apologize for being so verbose.. trying to learn as much as possible, as quickly as possible.. gotta get this new nas going..

 

[oneplane]

It does indeed not work like that. In essence the thought pattern for ZFS would be 'how safe would I like my data to be' and then use one of the well-known configurations to make it happen. That means you don't really have to think much about how ZFS does redundancy or parity, only how much failure you would like to be able to handle.

 

[jcizzo]

It does indeed not work like that. In essence the thought pattern for ZFS would be 'how safe would I like my data to be' and then use one of the well-known configurations to make it happen. That means you don't really have to think much about how ZFS does redundancy or parity, only how much failure you would like to be able to handle.

yeah! that's a good way of looking at it...

i'm just really taken aback by how much stress and ultimately how dangerous a resilver can be.. never knew..

so back to my original question/theory: i understand the benefits of raidz2 of course.. however, if that's not a possibility, is doing the whole "raidz1 with spinners" plus a hot spare SSD possible? or is that asking for trouble?

general hotspare question: when a drive dies and zfs automatically rebuilds the raid using the hotspare, does the hotspare become permanent or is it returned to the pool as a hotspare once the dead drive is replaced?

Thanks you all for your time and patience!

 

[ericloewe]

what could be cool (if it were possible) is the 5 spinners for storage and 1 (or two mirrored) ssd's for parity.. but i don't believe any of this works in that manner.. if it did, i'd assume that would alleviate a lot of the stress during a resilver.. correct?

No, the concept does not make any sense, which is why RAID3 hasn't been relevant in two decades and has been fully superseded by RAID5. Having a single parity disk creates asymmetric access patterns that degrade performance.

RAIDZ goes beyond that and allows for variable-sized blocks, so it makes even less sense to separate out the parity.

i guess i could do the 5 disk raidz2, but again, i'm starting to run outta space.. plus i figured raidz1 across 5 drives would be more performant

Any difference is going to be trivial. If you need more performance, use mirrors and/or SSDs.

What really matters is that a single RAIDZ1 vdev plus a hot spare is the worst of both worlds. The drive is subject to wear, it's consuming power, but it still needs to resilver to do anything useful. The only situations where hot spares make sense are if you have multiple vdevs (the spare can service them all) or want better-than-RAIDZ3 reliability.

is doing the whole "raidz1 with spinners" plus a hot spare SSD possible? or is that asking for trouble?

You could, given an appropriately-sized SSD. Seems like an expensive solution to the problem. Again, the hot spare is a bit weird, since you could just do RAIDZ2, but I can see the argument if there's an SSD in the mix, given write limits.

i'm just really taken aback by how much stress and ultimately how dangerous a resilver can be.. never knew..

I absolutely hate the "resilvers are so stressful" line of reasoning. It's not backed by any real evidence, just weird hypothetical edge cases. I think it must have originated from people whose systems were in shambles and didn't realize it until a disk failed outright, by which time other issues had accumulated and ruined their day.
What is far closer to reality is that losing a disk in a RAIDZ1 or two-way mirror leaves you with zero margin for error. That other disk had a bad block and corrupted some data? It's gone. Second disk died? Sucks to be you. Yes, all these factors can be mitigated, but one extra disk really does not represent a huge cost for the massive improvement in reliability and peace of mind.

Sidenote: you don't have to remove a disk to replace it: as long as you can attach both new and old disk, the process can be done with the old one in place. I wouldn't recommend relying on it, but aggressive early replacement of disks with signs of trouble developing could be used to mitigate some of the risk of RAIDZ1, if you're in a really tight spot.

 

[reasonsandreasons]

I do agree with @ericloewe's point about resilvering, though I think that most of what's meant by "resilvers are stressful" is consonant with it. Resilvers are not uniquely stressful compared to regular pool access, but they're still a consistent, fairly high long-term load on disks that are probably around the same age. No margin for error in z1 doesn’t mix well with that. There's also probably some imprecision around the term, since resilvers are often emotionally stressful. I was definitely sloppy with my usage, though, and I apologize for giving @jcizzo the wrong impression.

Some other thoughts:

• If this is for media storage, avoid worrying about speed and focus on fault tolerance. Even high-bitrate content is on the order of tens of megabits per second; my 4k60 rip of Gemini Man (a bad movie but a useful example) caps out around 90Mbps. Any modern pool will be able to get you what you need unless you have an absolute ton of users.
• If you already have 10TB of media, 20TB raw isn't really enough storage, especially if it's off-site. You're going to be at 15.4TB usable with a five-drive RAIDz1 config and if you leave 20% free (generally good practice) you only really have 12.3TB to play with for new content.
• If you're the kind of person who already has 10TB of media, you will want more. Good article on this here; the whole site is worth reading if you're getting into ZFS. If I were building I'd aim to be at max 50% full when the system is built, ideally with free bays available for more disks (but I'm a mirrors person, so this is less applicable for RAIDz).
• When you do want to increase your storage with a RAIDz configuration, that means a long, annoying process of pulling one drive, replacing it, resilvering, pulling another drive, resilvering, and continuing until all the disks are replaced. Then you'll see your new capacity. This will take a while and require a lot of manual intervention; avoid this for a machine that's not your home or place of work.
• Mirrors solve this by just letting you add more vdevs in parallel with the existing ones to use the storage right away. You end up with imbalances in the pool but given enough time you can fix that. Until RAIDz expansion comes (read this in the way you'd read a statement about fusion power), mirrors are much easier to manage as a home user pulling disks together.
• Having a backup mitigates some of these concerns; instead of doing the resilver dance you can just destroy the pool, replace it, and copy the data back. However, backup gets fraught as your storage needs go up. Cloud backup for Linux and BSD basically requires a service like Backblaze B2, which will run you $50 a month for the 10TB you already have. Local backup is an option, but after the 20TB mark or so you're going to need an entirely separate pool, and at that point you might as well have a second dedicated machine to house it and regularly receive snapshots. You should do this (your Plex library is important and it would be sad to lose it!), but having a backup isn't really a good way of reducing total costs.

 

[jcizzo]

@ericloewe and @reasonsandreasons

you two are f-ing AWESOME!! i TRULY appreciate the time you took to help me out.. this was absolutely great!

response to some of the thoughts you both shared:
@ericloewe
" What really matters is that a single RAIDZ1 vdev plus a hot spare is the worst of both worlds. The drive is subject to wear, it's consuming power, but it still needs to resilver to do anything useful. "
-> this is precisely the reason i wanted to do a raidz1 with spinners (because the cost/GB is still the most reasonable), with a single SSD (of 5GB or greater) as the hot spare, because the SSD wouldn't use any power while idle (compared to a spinner, plus essentially no heat), so no wear, and during a rewrite, it would be able to handle all the bandwidth the remaining drives could throw at it without issue, speeding up the process. yes, the SSD is more expensive than an HDD, but for that emergency.. IF i had the drive space for another 3.5" spinner, we wouldn't be having this conversation.. would've gone raidz2 without question. the SSD was a theory i concocted and, if possible for my use case, would've made a lotta sense it was just a thought and an opportunity to learn something.

reasonsandreasons
regarding your thoughts on additional storage; yes i thought of that too.. i'm always gonna need more and yes, losing my plex library would suck.. however i kind've thought that by imposing a hard limit on storage, it would force me to do house-cleaning and get rid of the stuff that no one watches.
regarding backup with an external drive, yes i was looking into that as well... figuring i'd just keep a copy air-gapped on an external drive plugged into the USB port of the plex box..

the last point is, i ultimately am/was trying to make this run on fairly low power. the more drives i have spinning, the more juice it's gonna take. plus, i'm trying to keep it all in the confines of an ATX case (with 3x5.25" drive slots in the front). it's a very simple, humble nas, that has been over-complicated by yours truly.

thanks again!

 

[BackupProphet]

You can put a spinner to sleep, it consumes like 0.1-0.5watt/h

 

[oneplane]

Perhaps this makes it easier to decide, because it's all about what's important to you:

1. You want some storage, so you get some storage
2. You don't want to have your storage go offline if a drive fails so you get an extra disk for redundancy
3. You don't want to immediately have to take action if a disk fails, so you add another disk for more redundancy

If you were to do something else, i.e. add a disk but not add it for redundancy, it's somewhat pointless. Redundancy-after-the-fact with a hot-spare in a single-redundancy setup doesn't make it more redundant or more reliable.

Personally I'd sacrifice some space and go for double redundancy.

 

[reasonsandreasons]

You can put a spinner to sleep, it consumes like 0.1-0.5watt/h

There's a script to do this for TrueNAS, which would even let you configure the hot spare to spin down while avoiding sleep on the other drives.

(Do some reading on if you want spindown for your whole pool--it might do more harm to reliability than it saves in power. Not sure how much of that is sysadmin superstition, though.)

 

[jcizzo]

There's a script to do this for TrueNAS, which would even let you configure the hot spare to spin down while avoiding sleep on the other drives.

(Do some reading on if you want spindown for your whole pool--it might do more harm to reliability than it saves in power. Not sure how much of that is sysadmin superstition, though.)

yeah, i don't wanna put the whole pool to sleep. i'd rather keep the disks spinning while the system is on. we've had a bunch of old pcs at work that had spinners and had been running for years and years.. when they were powered off fully, they wouldn't restart because the drives finally bit the dust.

regarding double redundancy, i completely agree and may just have to bite the bullet.. i just figured the 5 x3.5" spinners in a raidz1 with an ssd as a hot spare was an elegant alternative to 5 spinners in a raidz2 because of 1) lower power usage, 2) theoretically faster rebuild times, 3) achieving more storage from 5 disks..

 

[bonox]

my only situation for using a hot spare instead of a parity drive is to cover multiple vdevs. In a 25 bay disk shelf for example, I'm running 3x 8disk Z2 vdevs and the last slot is a hot spare which can be used in the event of a failure in any of the three vdevs. No point having a fifth wheel that's not carrying any load if you've only got a single vdev imo.

 

[mr44er]

Think about a higher redundacy as "yes, it protects my data better" and not "no, that eats up all the capacity". Your data is the only true value, it's not important if the machine is fast or the disks are shiny.
Also think about raidz2 or 3: in the event of a disk failure, the 'spare' immediately runs.
If 5 slots with raidz2 gives not enough netto capacity, then you need a bigger case.


If you want to protect your gold nuggets, you buy a thick and bulky safe. You don't complain how it's eating up space in your room or think about the cheap model with the 2mm aluminium case and pepsi logo.