Why I love ZFS and Napp-It

[MatrixMJK]

Just a note to the reliability and survivability of ZFS and the great tool Napp-It that sits on top.

TL;DR... ZFS was recoverable even after 7 out of 16 drives had been kicked out of my array due to overheating.

In my garage at home I have several servers and disk shelves and other gear making up my home lab. This also includes a server and VMs that host media for playing to STBs and portable devices around the house.

At some point over the weekend some papers fell or blew in front of the media server and stuck to the front of the drive bays, blocking pretty much all the air flow into the front of the server. It was like that for at most an hour before I noticed it. By the time I found the problem 7 out of 16 drives had gone offline due to over-temps and ZFS kicked them out of the storage pool. The pool is 16 x 4TB SAS drives in a SuperMicro chassis and has been running for more than two years.

I shut it all down and let it cool off for a while. I powered up the system and the HBA saw all the drives and no error states on them. ESXi came up OK and I started my OmniOS/Napp-It VM that has the HBA passed thru to it. ZFS still had the pool degraded, but I had seen this before on other systems (not with this many expelled drives however). In the Napp-It GUI the drive list only showed 9 drives and looking at the 'Initialize' menu did not show any other drives available. This was worrying me because the drive controller saw all the drives. Going to the command line, the 'format' command only saw 9 nine drives as well. Since I did not have a current backup of all that data, I was really starting to sweat.

On my ESXi server I had installed the latest Napp-It 'OVA' to test, so I thought I would pass the HBA to it and see what it could see. Thankfully it saw all 16 drives, and said I could import the pool, and the pool did not have any missing members. I configured Napp-It etc. as best I could then successfully imported my pool. After fixing some permission issues all is well and the family is happy again watching TV and movies as they please.

I am still not sure if I could have done anything to get the drives to show back up to the original machine. After importing the pool to the new machine I exported it back out, and tried to import it back into the original VM, but it was still ignoring the 7 disks it had kicked out like to OS didn't see them any more.

I know other RAID system may also be able to recover from a situation like this, but ZFS has never let me down at home or at work. There we have two 2PB systems using ZFS as a back end to Luster and have never lost a bit of data. I have had hardware RAID failures that several times forced me to go recover from backups. I am a firm believer in software RAID and especially ZFS.

Thanks to all those that have built and tested ZFS and also to @gea for the great interface to it!

 

[Derf]

Cool story, but 16 x 4TB drives, does anyone really need that amount of data backed up for personal use? Seems like you just have a lot of movies you are holding for on-demand streaming. Considering your hardware cost, time spent for configuration/recovery, electric bill, etc, wouldn't Netflix/Hulu/HBO Now/etc be a cheaper solution?

 

[MatrixMJK]

Certainly not ALL of it needs to be backed up. All the important data is backed up in a couple more places. This stores more than just movies and other media. The backups are to other drive arrays that are powered off most of the time unless active. There are certainly cheaper ways to accomplish some of what these systems do, but I would not get the experience from just using streaming services. I get a lot of benefit from building my own.

This does use quite a bit of power but it also stores stuff for my work and learning, since in my real job I do very similar things. A lot of this is more an investment in my profession, it just happens to be a good place to store fun stuff too.

Recovery scenarios are good experience for this profession.

 

[T_Minus]

Cool story, but 16 x 4TB drives, does anyone really need that amount of data backed up for personal use? Seems like you just have a lot of movies you are holding for on-demand streaming. Considering your hardware cost, time spent for configuration/recovery, electric bill, etc, wouldn't Netflix/Hulu/HBO Now/etc be a cheaper solution?

Why do you care what he has or how much storage?

I think it's kind of arrogant of you to make comments about others storage with 0 knowledge of their setup/utilization. Maybe he doesn't have internet 24/7 for those solutions? Maybe not enough bandwidth?

Regardless the post wasn't about his storage, asking for criticism, etc... it was about recovery, and ZFS / Napp-It.

Sorry but your post really rubs me the wrong way. Maybe it's simply how you come across with: "cool story, now let me tell you how to do it better, my way".

 

[ttabbal]

Not to mention, with data caps becoming "normal" .. sigh... My kids can use a LOT of bandwidth watching the same show over and over and over again... Streaming loads a copy from the internet every time. Local copies don't hit the net or the caps.

 

[gigatexal]

Cool story, but 16 x 4TB drives, does anyone really need that amount of data backed up for personal use? Seems like you just have a lot of movies you are holding for on-demand streaming. Considering your hardware cost, time spent for configuration/recovery, electric bill, etc, wouldn't Netflix/Hulu/HBO Now/etc be a cheaper solution?

Hey man we are nerds. Let the man build backblaze in his garage if he likes. And sometimes offline content is so much better than streaming.

 

[Derf]

Yup, I'm a nerd too, super nerd, that's why I'm here and running OmniOS instead of WHS or some Linux distro, or FreeNas on my own hardware and not some Synology NAS. And I got the point of the post being about recovery ability of ZFS. I actually read his entire post.

That said, the Internet is getting waaaaay too sensitive these days.
If someone says "look what happened" and I make a post asking "why do you do that" why are there multiple people jumping on me about "who cares" "how arrogant of you to wonder that" and "let him do whatever he wants, we're nerds" and giving hypothetical reasons about data caps on broadband internet and anecdotal stories about kids rewatching stuff. My kids rewatch stuff too- the same movie over and over and over. That 1 movie takes up maybe 10GB, not 50Tb.
7 disks out of 16 is almost 50% of disks down. Should I extrapolate this survival case to my storage pool of 4 disks being able to survive if I lose 2? And if not, what is the big take home message here? Besides the reverse trolling happening on forums these days.

 

[Derf]

Why do you care what he has or how much storage?

I think it's kind of arrogant of you to make comments about others storage with 0 knowledge of their setup/utilization. Maybe he doesn't have internet 24/7 for those solutions? Maybe not enough bandwidth?

Regardless the post wasn't about his storage, asking for criticism, etc... it was about recovery, and ZFS / Napp-It.

Sorry but your post really rubs me the wrong way. Maybe it's simply how you come across with: "cool story, now let me tell you how to do it better, my way".

By the way, I care not about what he has or how much storage, but WHY he has that much storage.

How is it arrogant to ask why he has that setup?

If you re-read my comment, I'm not saying "cool story, now let me TELL you how to do it better, my way", I'm ASKING, "cool story, WHY isn't the alternative of paying for streaming services not better than A SUGGESTED ALTERNATIVE?"

Another point that I could be asking is, why is it preferable for someone to buy all this hardware, set it up, maintain it, upgrade it, etc, taking into account all the money, time, stress, etc...
Not to mention
A) all the time spent ripping all of your legally purchased content so it's available for streaming on demand, or
B) pirating all the content requiring such a large storage pool, which does allow storage requirements to grow, but ethically is dubious.

As opposed to paying for streaming services.

/flame suit on/

 

[K D]

Folks... ZFS... Napp-it... Let's stick to it. This post was a good feedback on two pieces of software.

Moving to ZFS from hardware raid recently I have been wowed by the portability. I have literally swapped different flavors of zfs implementations - freenas,napp-it,Zol about 30 times with thr same pool of drives without a hitch. Especially running it virtualized, just connecting a JBOD to a different head and being able to get running again in 2 mins has been a different experience for me.

I do run a 24 x 4tb pool backing up to a 16x8tb pool.


Sent from my iPhone using Tapatalk

 

[gigatexal]

Back in the day I had 24x3TB drives running ZFS for no reason other than I could.

 

[T_Minus]

Yup, I'm a nerd too, super nerd, that's why I'm here and running OmniOS instead of WHS or some Linux distro, or FreeNas on my own hardware and not some Synology NAS. And I got the point of the post being about recovery ability of ZFS. I actually read his entire post.

That said, the Internet is getting waaaaay too sensitive these days.
If someone says "look what happened" and I make a post asking "why do you do that" why are there multiple people jumping on me about "who cares" "how arrogant of you to wonder that" and "let him do whatever he wants, we're nerds" and giving hypothetical reasons about data caps on broadband internet and anecdotal stories about kids rewatching stuff. My kids rewatch stuff too- the same movie over and over and over. That 1 movie takes up maybe 10GB, not 50Tb.
7 disks out of 16 is almost 50% of disks down. Should I extrapolate this survival case to my storage pool of 4 disks being able to survive if I lose 2? And if not, what is the big take home message here? Besides the reverse trolling happening on forums these days.

Trust me, I'm not sensitive.

It's your attitude. I'm leaving it at that. If you have any questions beyond that then send me a PM directly.



I don't know about the OP but I buy drives on sale based on value/GB so I may have a BUNCH extra for a couple years to 'catch up' but I know I'm good for another 3 or so too

I think right now I'm using 1.5TB out of 18TB usable

 

[ekke]

Cool story, but 16 x 4TB drives, does anyone really need that amount of data backed up for personal use? Seems like you just have a lot of movies you are holding for on-demand streaming. Considering your hardware cost, time spent for configuration/recovery, electric bill, etc, wouldn't Netflix/Hulu/HBO Now/etc be a cheaper solution?

One usually have both.

Skickat från min FRD-L09 via Tapatalk

 

[Chris Moore]

does anyone really need that amount of data backed up for personal use?

Who are you to even ask if he needs it? That is like the libtards that ask who needs a rifle or pistol. It is my right to have it if I want it and can afford to buy it and I don't need to need it.