I need a 'Redundant Array of Inexpensive Tape!' (LTO tape parity set creation)

Redundant Array of Inexpensive Disks - based around the idea that one disk in a set can fail or is seen as a failure risk, and more space efficient than straight mirroring while potentially being able to tolerate more failures. (especially the 'wrong two' mirrored drives failing)

I need to apply the above - to LTO Ultrium tape and i'm trying to figure out HOW. :)

File level or block level is possible - but i'd vastly prefer file level if I can simply because then I don't have to restore every single tape in a set to see things, and I can individually pull files when demanded. This probably means some kind of erasure coding on the file level - i've investigated PAR like used on usenet but it's not designed for terabyte level workloads or sets. Even creating I think it was a 100gig fileset gave me a finish time of over 30 days when I tried once. I can't process for years just to create one set.

Backups and mirroring would STILL be done because i'm aware RAID is not a protection against site disaster. This is just a way of improving the recovery safety of something that may sit on a shelf for decades and with murphy's law being what it is, i'll get the exact same two tapes to break somehow in each set. It's closer to preparing for "a disaster happens at site A, so I get my box of climate controlled storage tapes from site B and begin restoring - and one of the tapes breaks!" At which point I either spend thousands for data recovery, or I cleverly packed away at least one parity tape for $20-100 bucks as insurance.


Also since were talking tape, does anyone know what happens if there's a bad file read on an LTO tape restoration? Does it stop the whole tape recovery, do you lose the file it was trying to restore while moving to the next one, or is there a way to make it just return a null value EVEN IF that file will be bit-corrupt because I can then repair it with a PARity set loaded later? I'd rather have a corrupt file than no file I mean, just marking it as corrupt and letting me decide what to do with it - i'm aware that may be up to the backup software and not innate to the format.

My ideal is files are just on tape in LTFS format, I literally pull the file I need, if it has a few corrupt bits I dont care because I then pull the next PARity files off the same tape and fix it. Or if a tape breaks I have a separate tape set set of parity files that backs up everything even if it requires every other tape in the set to be loaded onto a server for reconstruction-recovery.
 

cesmith9999

Well-Known Member
Mar 26, 2013
1,214
365
83
Short story, you don't. Tape is a VERY different beast. meant for streaming data on and off tape. it is hard to sync different tapes to get that RAID 3/5 level of parity. It is just (orders of magnitude) easier to have 2 different backup tapes/sets (kinda virtual RAID 1) than to try and apply any raid 3/5 like redundancy to a tape library.

If there is a bad spot on the tape the backup software usually flags it as bad and continues on to the next file.

If you are afraid of bit rott. you need to be looking at other solutions that do not involve tape. this is where cloud storage comes in. where you pick a provider and let them do the tape upgrade in the future. the largest issue I have with tape backups where the media sits on a shelf forever is that you have to have a tape library that has compatibility with the older tapes handy. and the same (or compatible) software that you wrote the backup with to get the data back off the tape.

Chris
Who has watched too many backup bit rott while sitting on a shelf and the tape libraries died...
 

cesmith9999

Well-Known Member
Mar 26, 2013
1,214
365
83
and I am involved in removing 2 VTL and a real tape library... all of our backups are now going to the cloud...

Chris
 

Scott Laird

Active Member
Aug 30, 2014
257
102
43
In theory, you could use something like zfec/zfec at master · tahoe-lafs/zfec · GitHub to break a single backup file (.tar, etc) into erasure-coded chunks, and then back up each chunk onto a different tape. You'd only need N of M chunks (you pick N and M beforehand) to get your data back... assuming that you don't lose track of which tape holds which file, and don't run screaming just trying to manage the mess.

Tapes are a pain in the neck for small-scale storage. LTO6 tapes aren't *that* much cheaper per TB than hard drives, and that's before you pay for the drives. If you're looking at multiple PB, or you have specific legal archiving requirements, then maybe tape makes sense, but they're too expensive, slow, and trouble-prone for general use.

Try pricing out cloud cold-storage options (pay attention to bandwidth costs!) and writing archive data out onto a few replicated HDs and storing them offsite, then compare that to the cost of dealing with tapes. Either way you avoid the up-front cost of a tape drive (or library) and your costs will scale closer to your actual usage.
 
I FOUND A WAY!! :D

I am still investigating the specific details, and I will be posting separate related topics on the subject, but a program called SnapRAID seems to be capable of doing what I want. (either as it is or possibly with small modifications) I will be trying to design around and build a test system soon.

SnapRAID works by having a set of data disks, and separate parity disks (from 1 to 6). It creates RAID style redundancy, but creates it OVER the file system, therefore the parity recovery information is saved in a normal file! My plan is to simply write this parity file out to Ultrium tape (along with the data volumes of course) and make matched tape sets with multiple data tapes (which are directly readable via LTFS - SnapRAID does not alter any existing data's format in any way) then separately writing out the parity tapes.

If a tape ever breaks (and the mirror set is not available which is more convenient, or the mirror breaks) I have to copy all data tapes over to a server properly configured - including the original parity tape (or two or whatever) and then use SnapRAID's functionality to restore the missing data. The people on the SnapRAID forum suggest I should be able to do this even though nobody has done so before. (and obviously I will TEST this before I rely on it!)
 

Evan

Well-Known Member
Jan 6, 2016
3,144
530
113
In theory it should work. Speed wise I can't imagine it being possible.
The wear on tapes and drives would be crazy.
Simply just make 3 copies or whatever.
 

Blinky 42

Active Member
Aug 6, 2015
568
203
43
44
PA, USA
I would still highly avoid the RAIT model - if you want protection for the data on tape, make multiple copies of each tape and store them in totally different offsite locations. It also would defeat the advantages of using LTFS to make the tapes easier to transport and use individually.
When used as the archive medium, you should be writing the tapes in a way that you have the best chance of easily making use of the data several years down the road.

Building up a fragile custom solution that only works when you have a set of drives on a single server is going to be a maintenance nightmare. If the building burns down that has the server and set of LTO drives in it, then to make use of all your offsite backups you will need to recreate that setup somewhere else with new hardware or play serious games by reading in all of the tapes from a single set onto a staging area on hard drives, then assembling the raid array out of those "images" of the tapes on the drives and pulling out that data you want.

The complexity I would focus on is if you are going to be dealing with files that are larger than can be stored on a single tape after the hw compression in the drive, come up with an internal standard way to split those files that is multi-platform friendly. Then you can pick 1 or 2 tapes as needed to get the materials for a project and take them with you or send them to a colleague who can mount the tapes using LTFS and pull out the files to work with.

Don't forget - LTO-7 only does 6TB raw. Tapes are going to be your long term archive and backup medium, not your reference it frequently to get assets from medium. I would invest in sets of 8TB (or the current TB/$ sweet spot) drives and use that in NAS or even external drive box like the Drobo STH did a review on this week Drobo 5C Self-Managing USB-C Storage Solution Review
 
In theory it should work. Speed wise I can't imagine it being possible.
The wear on tapes and drives would be crazy.
Simply just make 3 copies or whatever.
It's okay, it doesn't have to work WELL, it just has to work AT ALL right now. :) This is not intended for production - this is intended to store large amounts of data that might need to be worked on a few times, but in general is just going to sit until we have the online NAS storage to mess with it.

At most i'm expecting a dozen extra tape loads for messing about when the tapes are designed for well over a hundred. As soon as it's more than a very rare occurance it's obvious it's going to have to be loaded onto the NAS and left there to be worked on.


I would still highly avoid the RAIT model - if you want protection for the data on tape, make multiple copies of each tape and store them in totally different offsite locations.

Building up a fragile custom solution that only works when you have a set of drives on a single server is going to be a maintenance nightmare. If the building burns down that has the server and set of LTO drives in it, then to make use of all your offsite backups you will need to recreate that setup somewhere else with

The complexity I would focus on is if you are going to be dealing with files that are larger than can be stored on a single tape

Don't forget - LTO-7 only does 6TB raw. Tapes are going to be your long term archive and backup medium, not your reference it frequently to get assets from medium.
Thanks for your suggestions, i've already given all this consideration. (not that I dont appreciate the asking of questions or analysis - I dont want to be in an echo chamber I mean) Mirroring is planned ALREADY - part of the reason for the parity tapes is to provide an alternative way to replace a single (or more) busted tape in a set for minimal extra cost.

The plan is to have two readily available mirror sets, and at least one set of tapes at a remote site (later planning two). If a tape mangles on load, I grab the mirror. If that also mangles I either drive to the remote site or have to restore all the tapes in the set plus grabbing the parity tape, and running a SnapRAID restore on those volumes - whichever is more convenient. If the tapes are cross country loading them all to the server is probably more convenient!

It gains me protection at minimal extra cost - it is not meant to replace mirroring. Especially if say both sets of tapes on site even in a fire safe were lost, and I have only the one set of remaining tapes and it's parity tape remaining. I'd bite my fingernails to nubs watching each tape slowly load otherwise - the parity gives me a One Last Chance to recover everything in a single dataset despite one or more broken tapes. "Belts and suspenders" for the guy really concerned about keeping his pants up.

Whether suggested or not suggested I plan to try it as long as the SnapRAID people can get me started anyways. If my experience works i'll be sharing it if anyone else finds it useful.

I wont be dealing with any files over even 2.5 terabytes right now so i'm not worried about monster files yet.

The plans for a monster NAS are simultaneously being explored with a scale out of 42 data drives right now. Because of my unique situation I don't know whether i'll need "more cheap data" (migrated straight to Ultrium) or "more workspace data" (multiple volunteers processing video for the project), but I have a scale out plan for each case whether i'm buying tapes or buying disks.
 

cesmith9999

Well-Known Member
Mar 26, 2013
1,214
365
83
It's okay, it doesn't have to work WELL, it just has to work AT ALL right now. :) This is not intended for production - this is intended to store large amounts of data that might need to be worked on a few times, but in general is just going to sit until we have the online NAS storage to mess with it.

At most i'm expecting a dozen extra tape loads for messing about when the tapes are designed for well over a hundred. As soon as it's more than a very rare occurance it's obvious it's going to have to be loaded onto the NAS and left there to be worked on.
Just build out your NAS. you will not waste your time on something that you are not going to use in the long run.

Chris
 
To the last two - the tape preparation will definately be used in the long run. There is no need to have petabytes of work data spun up on drives using electricity, creating heat, and needing replacement every 4-5 years that's not being done anything with. Ultrium is designed for long term 30 year archival with reliable recovery - the ONLY flaw is "if a tape breaks". Mirrors help, but parity tapes are a "belt and suspenders" solution for if you have to be extra sure you are protecting the data. Tapes are more robust for mailing. More suited to sitting untouched for years whereas drives bit rot in place and mechanisms may refuse to spin up again. (i've seen all these errors)

The NAS is for workloads. The tapes are for backups and deep archival of data that cannot be thrown away but which doesn't justify excessive money in maintaining easy immediate access to either. Both will scale up according to need. If the NAS ever sees 300TB I will probably have 5 petabytes of tapes on the shelf! This is what other pro level people are doing with digital video in long term storage already - all i'm doing is adding parity tapes to whats already a known and recommended best practice.


I'm aware of backblaze. It's just the cost of upload bandwidth cannot make uploading terabytes per day feasible. Cloud is not very useful to my use case.