"Raid Wars", or AKA "Raid Level 5 Considered Harmful"

[Joseph Nunn]

I'm starting a thread to discuss the relative merits of the various RAID levels and their use, so others who are considering how to organize their arrays may benefit from the collective knowledge pooled. Welcome are links to any kind of academic research, benchmarks, or other articles exploring the relative merits of the RAID levels. Also welcome are links discussing software vs hardware RAIDs, filesystem RAIDs (raidz), or any other stratagem by which separate storage elements are combined into a smaller number of images and their use cases.

As RAID levels used by an individual can be somewhat of a religious topic, often being a personal choice, lets endeavor to rely upon reason rather than emotion when comparing the merits, whatever our personal RAID choices have been.

The above shall hold true for the remainder of this thread excepting discussions about RAID 5, where there is no good use case to be made. No one expects the RAID 5 Inquisition, so I will begin with links (garnered without attribution from another thread) discussing RAID 5's obsolescence and exploring the current state and predicted future of RAIDs.

Please review the following, and comment:
Triple-Parity RAID and Beyond - ACM Queue

Also:
When No Redundancy Is More Reliable – The Myth of Redundancy | SMB IT Journal
BAARF

 

[NetWise]

Yeah. Now that I've read the article I see how the 'RAID5 is obsolete' is both stupid and not clarified.

With a URE one in 12TB, one shouldn't do it for raid sizes that large. Sure. I'm totally on board for that. -in situations where the rebuild has too much data to survive being degraded-

But say an 8 drive RAID5 made up of 450gb 15k's. You're looking at 5.4tb. That's JUST as safe today as it was years ago. Safer even with improved controllers and such.

It's not until you get above 11tb that it starts becoming obsolete.

Which means it's still a perfectly valid consideration for say 36x300gb drives. Myself I'd probably go RAID50 in that case.

To blindly say 'raid5 is obsolete' is pretty much saying there is only one good solution. Which is never true.


Sent from my iPhone using Tapatalk

 

[T_Minus]

The RAID5 is obsolete comes from the statistical probability of a 2nd failure based on capacity and URE, and with increased drive sizes it is in fact stupid, and clarified in well written articles as to the reasoning behind this.

I find it amusing the article actually states the probability of a 2nd drive failing is "low" and that after 1 failure a 2nd failure is not anymore likely... because rebuilding doesn't hammer the drives more than normal... ya right. Then take into the URE #s and capacity of the drives and it makes complete sense to never do RAID5 in an environment where you value access to your data, uptime, not-nagging boss/end-users, etc...

RAID is not backup it's insurance from down time and the value derived from no down time is unique to all of us. Some have kids that would complain, nag, and go crazy if their movies/shows were not accessible others start losing money per-minute sometimes thousands of dollars per-minute, etc...

If you have backups and don't care about down time then RAID1,5,6 etc... is absolutely useless to you and a huge waste of money.

 

[Patrick]

At triple parity, items like disk controller, PDB and etc failures become more important.

 

[frogtech]

Hate to be a little off topic but unRAID is one of my favorite implementations of a parity based array in that if the rebuild fails you still have access to data on individual disks. Does RAID-Z1,2,3 do this as well? Conventional RAID 5 and 6 would probably be respected a bit more if they offered you 'access' to your data after a rebuild failure like that. At least, coming from the people who shun highly against it.

 

[unwind-protect]

Stupid article.

The real questions are:
1) how many redundant disks do you have?
2) what is the expected rebuild time?
3) if you don't have hot spares, how fast do you insert a new disk?
4) what is the time to restore the array from backup?
5) how up-to-date is the backup?

You calculate the chance and duration of downtime from there and weight it against cost in disks, controllers, housing and power. For quick, small arrays raid5 can come out entirely reasonable.

Backup can also be very fast these days, e.g. ZFS snapshot send/receive.

Or as another mental model, people are often fine with 2-disk raid1, because it "extends the machine lifetime" - but the same people then lose their minds when a 3-disk raid5 comes along.

 

[EffrafaxOfWug]

T-Minus speaketh le truth in my personal opinion; I've personally lived through two RAID5 (and one RAID6) rebuild that ran into UREs and had to be rebuilt from clean and restored from backups. The article is talking bullock glands in that regard IMNSHO.

RAID buys you uptime; backups buy you mere existence in the manner of surviving lack of uptime on your RAID. Its up to your environment which is more important but both have a price, but on a personal level, backups are much much more reliable than RAID, so spend your money on backups before you spend it on RAID. If your RAID is super ultra-mutto-reliable supremo-specialingtons then put it on RAID1 or RAID10 which are super-fast to rebuild (straight dd of a disc rather than the whole parity read-write-rebuild fandango) and steer clear of unrecoverable parity RAID like one thou wouldst of a drunken mafioso nephew of an early friday evening; it's possible to tackle, but you shouldn't stake your life on coming out the better for it.

With the size of (platter-based) discs these days, along with the inherent not-increasing-by-an-order-of-magnitude-access-speed, personally I'd be avoiding RAID5 like the plague on anything bigger than 1TB, and looking on RAID6 like it was a possible super-flu. Number one consideration in my line of work is "how much it done cost if this data isn't available to User XYZ?" and number one consideration in my line of living at home is "how much it done cost if my good partner isn't able to watch Single Female Lawyer when the mood demands it"? The answer, in all cases thus far, has been "considerably more than the cost of a few poxy hard drives" so my advice would be, if you share your sphere with someone besides your good self, would be to stay the feck away from anything resembling a "I've got to restore from cloud honey! It'll take a week!" (IGTRFCHITAW for short) scenario. Because the good partner will give you that icy cold stare o'doom if you come in wearing that IGTRFCHITAW-face again, because if a partner can't trust you to maintain access to Single Female Lawyer... what can they trust you to do?

 

[NetWise]

I have no such wife issues. There's always Netflix and YouTube as alternates during rebuild/restore. Secondly, like an employer, they tend to veto the up front cost to avoid the failure, and thus, they can live with the resulting issues


Sent from my iPhone using Tapatalk

 

[Patrick]

I have my wife thinking a ceph cluster is a solid next generation plan. Easy to expand!

 

[Jon Massey]

Had an old 5-bay Thecus NAS with 5x 1TB disks shit its pants about 36h into a rebuild. Never again!

 

[Patrick]

Had an old 5-bay Thecus NAS with 5x 1TB disks shit its pants about 36h into a rebuild. Never again!

Yea I have the old Synology DS1812 running backup of backups duties for that exact reason.

 

[BackupProphet]

For cost effective massive storage I would rather go CephFS than RAID. When "rebuild" time is taking longer than a week you may risk running into a scenario where you have trouble to complete the rebuild. I've been there once and its friggin annoying.

 

[Scott Laird]

Backups are fun. Life usually gets less confusing if everyone stops conflating a bunch of different issues all together and simply looks at it as a form of risk management. How do you handle a disk failure? How do you handle a multi-disk failure? How about a power failure that kills multiple drives? A DC fire? Law enforcement confiscation of servers? Bad software that slowly corrupts the data on your disks? Deliberate attacks? Legal requirements to produce data from day X? Catastrophic bugs in storage software that can't be fixed in a reasonable timeframe? Storage platform obsolescence?

No single approach can solve all of those for all use cases. You always end up with some hybrid approach--RAID + tape + offsite replication, for instance. Or you say that "we don't care about recovering from case X." Ideally in writing, agreed on by all parties, if money is involved.

Frankly, RAID itself is frequently a mistake; you'd be better off with X machines each with one replica on a single disk then you'd be with a X-way redundant RAID array on one system. Managing this sort of thing is a pain in the neck, until you have the right tools in place.

For home use, I keep most of my data on ZFS, with the important stuff backed up via a cloud provider, and everything replicated onto CephFS, which I don't really trust enough to make it my primary storage. I'd love to get rid of ZFS, but I'm still not 100% comfortable with Ceph long-term. This scheme can cope with hardware failures and rapid corruption, but software that corrupts files slowly over time probably wouldn't get caught before backups expired. I can cope with that.

On the business side, I've ran services before that couldn't really afford to *ever* do a restore from backups. We estimated that it'd take ~18h to restore from tape given our best option for tape storage. We still did tape backups (and tested restores), but they were really just there in case of an utter disaster. Frankly, we couldn't really even afford to restore on-disk snapshots; there were tens of thousands of transactions occurring every second, and throwing away transactions would have been ugly and generated a bunch of bad press. My worry at one point was that a bad enough screwup on our part could torpedo global trust in "cloud" services. So we replicated our data *widely*, had very careful access controls, and used a storage format that didn't overwrite data in place, so it was possible to merge backups with live data and get reasonable results in most cases. And then tested everything that we could test to within an inch of its life.

 

[cheezehead]

I have my wife thinking a ceph cluster is a solid next generation plan. Easy to expand!

Developer or Linux Admin?

 

[ttabbal]

As a home user, my priorities are pretty simple. I want reliability, and, lately, I want more speed. Maybe I shouldn't have gotten 10gbe, when I was limited to 1gbe, it didn't seem like such a big deal... I also want easy expansion either with disk upgrades or adding disks. Either is fine. I do have backups, but I do NOT want to use them. That would cause no end of whiny children to deal with. So redundancy is a requirement. RAID0 not allowed.

I have transitioned from 2x6-disk raidz2 to 6 mirror pairs (striped), and will probably add a few more now that it's been proven out, just to boost speeds up a little. The sucky thing about raidz2 for me, was that to get more space I needed to add another vdev, or replace every disk in a vdev one at a time with a bigger one. And that the speed, with 2 striped, could be better. I will note that I have never had a reliability problem with raidz2. I've had the array running for about 8 years, lost plenty of drives, even had an emergency when one drive died completely and I had bad sectors on another. The array rebuilt fine after a trip to Best Buy to get "any disk big enough" to rebuild ASAP. The emergency was my own fault, I got lazy and neglected the disk I knew had bad sectors. Never again.

I considered going to raidz3, but decided that my new chassis with 24 slots and some cheap 2TB drives gives me some flexibility. And as drives get bigger, they haven't been getting faster. So rebuild times are getting unreasonable for my uses. Mirrors rebuild fast and have less performance loss while doing so.

With frequent SMART tests and ZFS scrubs, I detect problems before they cause data loss. Will they fail me eventually, probably. That's what backups are for. But the downtime causes enough irritation that I don't want to deal with it if possible.

The backup box may well get raidz1 *gasp*. It's backups, as long as I fix issues and re-backup, it's all good. I'm considering making it an off-site though, so I might need to do better than that just to prevent long backup times for the first upload. Thinking of that, anyone want to swap a couple TB backup space? We could set up an STH backup ring. Crashplan, rsync, zfs send, whatever...

 

[MiniKnight]

We could set up an STH backup ring. Crashplan, rsync, zfs send, whatever...

People have tried this in the past on STH.