Hello, this is my (second) post (first was responding to a help request) because someone on SmallNetBuilder suggested I come on over here because this either is more my style, or may be a necessary part of what I have to do.
My name "Twice Shy" should suggest to you that i've suffered serious data loss in the past which I am trying to prevent EVER occurring again.
Basically I need an Enterprise class solution. My problem is that I do not have an Enterprise budget. Even though some/much of the data i'll be working on, I would like to think, will eventually pay for itself, this is part of a bootstrapping startup that has to get by on a shoestring for YEARS.
I already had dozens of terabytes and lost probably at least a third of it. I'm sure alot of the rest is corrupt by now. My previous solution of "buy 3tb Seagate external USB's" when we needed data to stick on the shelf turned into a Total Epic Fail.
Basically put me and several others are going to or will be going to film school (not all at the same time) and have the opportunity to access hardware for free that we cannot readily pay to rent later, like shoot 8k video and motion capture which involves like 16x HD streams at 120fps at 1080p. We are trying to do a side projects with it ("our own movie" plus a bunch of shorts) which we hope to serve as demo reels, calling cards, and similar learning projects.
This is very data intensive. A Red Weapon 8k shoots at 300 megabytes a second per camera (and more than one camera for something like a stunt is the norm), the mocap studio uses even more. Terabytes per day is possible if we pull an overnighter or weekender in the studio when nobody is using it.
300TB is a number I pulled out of my rump and nothing that will be built overnight, i'm not seeking suggestions for tens of thousands of dollars of server parts. That budget doesn't exist. What does exist is a strategy of trying to slowly grow an archive with the minimum cost overhead over that of hard drives possible. I'm not wanting rack gear in the home just to have rack gear (cool as it is) and didnt want to play sysadmin but rather trying to solve specific problems with specific solutions.
Films like Rogue One have more than a petabyte of digital data for comparison but there's many things we can do if we HAVE to to cut down on the data stream like shooting lower resolution and reducing the number of takes saving only the best ones even at the time of capture. 300tb can shrink to 60tb or 30tb if it has to, it's more "it would be nice to save all you can of everything as much as possible" so instead of saying what I want is impossible please remember that this is a very flexible process.
The two biggest problems to solve things so far unfortunately are people saying "hire a professional" when none is available (I know other students with that background but they are busy getting paid when they graduate, they arent in the struggling suffering artist collective), or "you cant afford to do this at all" when I already admitted how flexible and progressive things are. Lacking a better metaphor this is something really like having a herd of deer pass by and you either bag what you can or you dont get another chance. Getting 1 is better than none, getting 5 better than 2, and getting 20 better than 12.
What I mean is i'm just trying to save and salvage as much as temporary opportunity makes available, just like I did before (when it was mostly lost and corrupted on seagate external drives), since I either make an attempt to use the resource or I dont, and if i'm going to do it i'm trying to make the best strategy possible at it to get as much as I can. The budget will be limited, and more TB I can save out of whats available the better because we wont have second chances to capture alot of footage - I dont want to spend money on fancy server racks when "more storage" could be bought instead.
All that said i'm wanting to share the first few early conclusions i've sort of already arrived at for commentary:
1) What i'm trying to plan around first off is a strategy that rapidly migrates incoming data to LTO Ultrium tape. Whether it's cost per gig, ease of storage without data loss, ability to mail data (wouldn't want to mail hard drives - i've seen what happens to the post!) robustly, this to me is a win. Once the cost of entry for a tape drive is exceeded of some tens of TB it only saves money from thereafter to say nothing of the long term storage costs.
2) I'm still researching the NAS/SAN storage options and nothing is off the table yet. We have to capture data now, but may slowly work on it even for years afterwards if there's no alternative/a quicker method isn't possible. What starts as a low watt learning NAS on gigabit will probably turn into a high performance SAN with SSD's and 4-40gigabit networking eventually. It takes as long as it takes - better to make progress than making none at all or give up not trying. Other than i'm trying to DIY and home roll a solution because i'm very interested in technologies like ZFS having already experienced significant silent data corruption, and also dollar for dollar i'll probably have a lower cost overhead DIY than buying some off the shelf answer.
3) Nothing is happening overnight and i'm the main person that has to learn and implement this because i'm the class nerd among the film geeks. Migrating incoming data to Ultrium tape will happen much earlier than setting up a successful NAS/SAN to work on/process the data. So i'll simultaneously be asking questions about immediate problems, as well as potentially problems a year off or even far off, just so that I know what i'm doing in the future/have the roadmap planned out. Even if I discover I cant afford a given solution for awhile yet for instance, or need to slowly collect hardware when opportunities present themself.
Sorry for this being so long and thanks for taking the time to read. I hope I can learn things and also at some point contribute knowledge back.
My name "Twice Shy" should suggest to you that i've suffered serious data loss in the past which I am trying to prevent EVER occurring again.
Basically I need an Enterprise class solution. My problem is that I do not have an Enterprise budget. Even though some/much of the data i'll be working on, I would like to think, will eventually pay for itself, this is part of a bootstrapping startup that has to get by on a shoestring for YEARS.
I already had dozens of terabytes and lost probably at least a third of it. I'm sure alot of the rest is corrupt by now. My previous solution of "buy 3tb Seagate external USB's" when we needed data to stick on the shelf turned into a Total Epic Fail.
Basically put me and several others are going to or will be going to film school (not all at the same time) and have the opportunity to access hardware for free that we cannot readily pay to rent later, like shoot 8k video and motion capture which involves like 16x HD streams at 120fps at 1080p. We are trying to do a side projects with it ("our own movie" plus a bunch of shorts) which we hope to serve as demo reels, calling cards, and similar learning projects.
This is very data intensive. A Red Weapon 8k shoots at 300 megabytes a second per camera (and more than one camera for something like a stunt is the norm), the mocap studio uses even more. Terabytes per day is possible if we pull an overnighter or weekender in the studio when nobody is using it.
300TB is a number I pulled out of my rump and nothing that will be built overnight, i'm not seeking suggestions for tens of thousands of dollars of server parts. That budget doesn't exist. What does exist is a strategy of trying to slowly grow an archive with the minimum cost overhead over that of hard drives possible. I'm not wanting rack gear in the home just to have rack gear (cool as it is) and didnt want to play sysadmin but rather trying to solve specific problems with specific solutions.
Films like Rogue One have more than a petabyte of digital data for comparison but there's many things we can do if we HAVE to to cut down on the data stream like shooting lower resolution and reducing the number of takes saving only the best ones even at the time of capture. 300tb can shrink to 60tb or 30tb if it has to, it's more "it would be nice to save all you can of everything as much as possible" so instead of saying what I want is impossible please remember that this is a very flexible process.
The two biggest problems to solve things so far unfortunately are people saying "hire a professional" when none is available (I know other students with that background but they are busy getting paid when they graduate, they arent in the struggling suffering artist collective), or "you cant afford to do this at all" when I already admitted how flexible and progressive things are. Lacking a better metaphor this is something really like having a herd of deer pass by and you either bag what you can or you dont get another chance. Getting 1 is better than none, getting 5 better than 2, and getting 20 better than 12.
What I mean is i'm just trying to save and salvage as much as temporary opportunity makes available, just like I did before (when it was mostly lost and corrupted on seagate external drives), since I either make an attempt to use the resource or I dont, and if i'm going to do it i'm trying to make the best strategy possible at it to get as much as I can. The budget will be limited, and more TB I can save out of whats available the better because we wont have second chances to capture alot of footage - I dont want to spend money on fancy server racks when "more storage" could be bought instead.
All that said i'm wanting to share the first few early conclusions i've sort of already arrived at for commentary:
1) What i'm trying to plan around first off is a strategy that rapidly migrates incoming data to LTO Ultrium tape. Whether it's cost per gig, ease of storage without data loss, ability to mail data (wouldn't want to mail hard drives - i've seen what happens to the post!) robustly, this to me is a win. Once the cost of entry for a tape drive is exceeded of some tens of TB it only saves money from thereafter to say nothing of the long term storage costs.
2) I'm still researching the NAS/SAN storage options and nothing is off the table yet. We have to capture data now, but may slowly work on it even for years afterwards if there's no alternative/a quicker method isn't possible. What starts as a low watt learning NAS on gigabit will probably turn into a high performance SAN with SSD's and 4-40gigabit networking eventually. It takes as long as it takes - better to make progress than making none at all or give up not trying. Other than i'm trying to DIY and home roll a solution because i'm very interested in technologies like ZFS having already experienced significant silent data corruption, and also dollar for dollar i'll probably have a lower cost overhead DIY than buying some off the shelf answer.
3) Nothing is happening overnight and i'm the main person that has to learn and implement this because i'm the class nerd among the film geeks.
Migrating incoming data to Ultrium tape will happen much earlier than setting up a successful NAS/SAN to work on/process the data. So i'll simultaneously be asking questions about immediate problems, as well as potentially problems a year off or even far off, just so that I know what i'm doing in the future/have the roadmap planned out. Even if I discover I cant afford a given solution for awhile yet for instance, or need to slowly collect hardware when opportunities present themself.Sorry for this being so long and thanks for taking the time to read.
I hope I can learn things and also at some point contribute knowledge back.
