Redundant Array of Inexpensive Disks - based around the idea that one disk in a set can fail or is seen as a failure risk, and more space efficient than straight mirroring while potentially being able to tolerate more failures. (especially the 'wrong two' mirrored drives failing)
I need to apply the above - to LTO Ultrium tape and i'm trying to figure out HOW.
File level or block level is possible - but i'd vastly prefer file level if I can simply because then I don't have to restore every single tape in a set to see things, and I can individually pull files when demanded. This probably means some kind of erasure coding on the file level - i've investigated PAR like used on usenet but it's not designed for terabyte level workloads or sets. Even creating I think it was a 100gig fileset gave me a finish time of over 30 days when I tried once. I can't process for years just to create one set.
Backups and mirroring would STILL be done because i'm aware RAID is not a protection against site disaster. This is just a way of improving the recovery safety of something that may sit on a shelf for decades and with murphy's law being what it is, i'll get the exact same two tapes to break somehow in each set. It's closer to preparing for "a disaster happens at site A, so I get my box of climate controlled storage tapes from site B and begin restoring - and one of the tapes breaks!" At which point I either spend thousands for data recovery, or I cleverly packed away at least one parity tape for $20-100 bucks as insurance.
Also since were talking tape, does anyone know what happens if there's a bad file read on an LTO tape restoration? Does it stop the whole tape recovery, do you lose the file it was trying to restore while moving to the next one, or is there a way to make it just return a null value EVEN IF that file will be bit-corrupt because I can then repair it with a PARity set loaded later? I'd rather have a corrupt file than no file I mean, just marking it as corrupt and letting me decide what to do with it - i'm aware that may be up to the backup software and not innate to the format.
My ideal is files are just on tape in LTFS format, I literally pull the file I need, if it has a few corrupt bits I dont care because I then pull the next PARity files off the same tape and fix it. Or if a tape breaks I have a separate tape set set of parity files that backs up everything even if it requires every other tape in the set to be loaded onto a server for reconstruction-recovery.
I need to apply the above - to LTO Ultrium tape and i'm trying to figure out HOW.
File level or block level is possible - but i'd vastly prefer file level if I can simply because then I don't have to restore every single tape in a set to see things, and I can individually pull files when demanded. This probably means some kind of erasure coding on the file level - i've investigated PAR like used on usenet but it's not designed for terabyte level workloads or sets. Even creating I think it was a 100gig fileset gave me a finish time of over 30 days when I tried once. I can't process for years just to create one set.
Backups and mirroring would STILL be done because i'm aware RAID is not a protection against site disaster. This is just a way of improving the recovery safety of something that may sit on a shelf for decades and with murphy's law being what it is, i'll get the exact same two tapes to break somehow in each set. It's closer to preparing for "a disaster happens at site A, so I get my box of climate controlled storage tapes from site B and begin restoring - and one of the tapes breaks!" At which point I either spend thousands for data recovery, or I cleverly packed away at least one parity tape for $20-100 bucks as insurance.
Also since were talking tape, does anyone know what happens if there's a bad file read on an LTO tape restoration? Does it stop the whole tape recovery, do you lose the file it was trying to restore while moving to the next one, or is there a way to make it just return a null value EVEN IF that file will be bit-corrupt because I can then repair it with a PARity set loaded later? I'd rather have a corrupt file than no file I mean, just marking it as corrupt and letting me decide what to do with it - i'm aware that may be up to the backup software and not innate to the format.
My ideal is files are just on tape in LTFS format, I literally pull the file I need, if it has a few corrupt bits I dont care because I then pull the next PARity files off the same tape and fix it. Or if a tape breaks I have a separate tape set set of parity files that backs up everything even if it requires every other tape in the set to be loaded onto a server for reconstruction-recovery.