Perhaps You need to reconsider the name of Your post, as it does not appear that is what you are aiming for, if I read your responses to the experienced folks are supplying responses for an "enterprise class" . Just an observation .....
Well I guess it's more like the Enterprise Class solution comes later. It's not that i'm against spending $400 for an enterprise drive that is $100 in consumer class - when the performance bottleneck/lifespan justifies it.
Right now the NAS is a cheap bit bucket. Even if we only produce seconds per day, the project gets done, just real slowly. Later, if kickstarter money comes in being impressed by the demos that took 10x as long to make as we wanted them get out, we migrate to a production workflow that is far more efficient so we can produce a minute or more of finished footage per day with multiple people working on it. Does that make sense?
Planning the end goal, and steps to get there, without some series of nightmare migrations where we find corrupted data, or a downed hard drive causes weeks of lost work. Plan the end goal - work backwards to the shoestring/bootstrap phase - take note of the clear steps required along the way that brought us there. If the NAS system were using at the end is the same as the beginning that's easier. If there are unavoidable steps, it's important to know when the jump points happen.
Assuming you are going to copy everything to tape for backup and archive, how big a pool of content do you need online for a specific project?
Off the top of my head I would consider:
- Offload/ingest speed - how much video do you shoot per day on a multi-day project and compare that to your pool of media for the cameras. The LTO transfer should be able to churn in the background while you are in the field, but you need have enough disk space to buffer the video pool waiting to be dumped to LTO
- Archive indexes + low res samples for context + full tape catalogue for your backup software (Personally I would want to be able to see stills from key scenes or low-res video samples of items in the library in addition to a robust naming scheme so I can get the exact tape I need for something vs restoring all the tapes for a project and then slogging through that)
- Redundancy + tolerance of data loss. You will probably start small here and grow as budget allows, but bare minimum of being able dump all source content to multiple tapes to keep a set offsite, and have enough tape head time to verify everything you write I would think you want to start with
- What setup you need in the field? Do you want to do offload from SSD's to a pair of small servers stocked with 10TB drives in the field and have them taken back to the shop for the rest?
You can split those up into different physical servers to get the ball rolling and then build up form there with a robust 40Gb+ network between the servers and build up your shared central video server pool over time but focus on the offload and robust LTO archive management now.
If you are the only IT/hardware guy in the group, and you want to do something other than be sysadmin for this setup all the time, spending $ on better solutions at the start saves your time maintaining the setup down the road.
Thanks for more great info btw. Right now the most flexible part of the workflow is the data coming in. If there is just no way to salvage 8k footage, it has to be saved as 4k to cut the data load in half. Turning what might be a 300TB array into 75TB overnight for instance. Or being aggressive instead of conservative with keeping alternative takes. At each step of the process there are adaptions that can be made.
My worst case estimate was taking up to several terabytes per day with all day multi camera shoots at max resolution raw. Anything that intense would have intermediate drives hooked to laptops. Multi day shoots at max everything werent planned - the "heavy" hardware is studio only. But coming home with another full 3TB drive and it's mirror could well happen.
If I thought it would get worse than this, that makes me prefer the "multiple NAS box" option so that instead of loading to one master server, we can just scale out/load it to multiple in parallel. Days of shooting could result in a few tens of terabytes yes, but nobody can take unlimited time off for a shoot so even multiday shoots were not too worried about going past a week at worst.
Buffering for LTO was the reason for planning D2D2T (front end video goes to one or more primary NAS boxes, which can have their own mirroring set up if the second stage is the bottleneck for tape writing, we might spend a week shooting video then several weeks swapping tapes but that's okay!)
Archive indexes - that's where we start talking multiuser/multicomputer. Setting up one or more transcoding servers to hammer the primary NAS boxes, downconverting them to basically modern workprint reels. We cant directly ingest or edit 4-8k at this stage, but we can transcode, make our cutlists, and show a nice 1080p demo before either seeking more funding, or doing the NLE the slow way offline.
Redundancy is currently planned everything has at least one backup/mirror at all stages in the process. When original raw footage is written to tape, it will be written to RAIT with parity sets set up in SnapRAID. (still researching this, the SnapRAID people suggest it's feasible to do what I figured out) Separate tape set archives of certain steps in the workflow as snapshots so we don't have to do over everything, ie different color grading passes, alternate cuts. Ultrium is supposed to verify on write - you can also do an extra verification pass. Since files written LTFS the extra verification pass should verify I can pull data straight off. Including of the parity tape. If an Ultrium tape breaks or dies, I rebuild the lost data by loading all the tapes of a data set, plus the one or more taped out parity sets to the server, to rebuild the volume - then rewrite out the volume. Eventually the two mirror sets (at least of original source footage) would become three, with one kept totally offsite for disaster recovery. (the parity tapes also assist this mirroring process as even the same three tapes in all data sets could break and I could rebuild it on the server)
Field setup - I haven't gotten that far yet. I'm hoping my thinking so far suggests I have a decent understanding of the total process without big gaps or holes though.
If I get one good primary NAS setup that I like that's high performance, the idea is I can replicate it across multiple boxes. Say one good 32TB NAS box might become ten (to give me 320TB at home), might become two smaller 15TB boxes in the video van just with 2.5" drives for more durability out there, but the same software... i'm trying to learn once then scale out.
Yes at some point performance and networking bottlenecks become issues - that's why i'm picking brains here, and putting Enterprise Class in the tagline. Because if I do need to get faster and faster, i'm wanting to have a map of where things need to go. On some level "it costs what it costs", on another I have to have several solutions basically fitting minimum possible budget, ideal budget, and at least one compromise level somewhere in between. We then know the figures that at least make the shoot possible in the first place, give us the best workflow/least wasted time and stress, and something in the middle based on the funding we can scare up. If we know we have the use of so and so's haunted house for a 9 day period in October - I can calculate the potential data use based on whether we can actually shoot 8k that long, vs falling back on 4k and such.
And what youre suggesting is exactly what i'm trying to do - first it's the minimum budget "straight to LTO" pipeline, just to make sure that data is backed up and integrity verified. The steps in between are about growing one or more NAS boxes, with varying level of consumer to enterprise class storage, having flexible plans to make the best use of what we have. Preferably using mostly the same software so I know I can handle it and make sure nobody loses any of their data, since by then i'm fully familiar with it, the workflow, and where to gripe that we need the next upgrades.
That last point is what i'm trying to feel out. It's not that I/we cant/wont spend money now/ever, it's about mapping out several ways of solving the problem so that we can make better decisions at the time. Given 10 ways to solve a problem, and 3 ways that i've highlighted as having unique strengths and weaknesses, it's easy to see the one best solution once we get there. Planning a path that only migrates data once maybe is better than four times learning four different systems. Planning something that works almost as well at 300TB as it does at 32TB is convenient to know. Changing our workflow so that it's slightly slower, but cuts the budget in HALF is super useful. Losing a little convenience to save alot of money is part of the game.
We are not super pro because we cant afford to be. What we are trying to do is punch well above our weight. We want to come up with stuff that has the pros go "WTF??!? How did they do that kind of work on THAT budget?!?" because were hoping that will lead to the funding to not have to cut corners anymore. And if and when we get to that point, i'm hoping the final step will be simply migrating all the data we have perhaps on hundreds of LTO tapes, now to some final monolithic server solution.
Working backwards the next to final solution is a still budget constrained "bridge solution" to that, working in steps all the way back to where I am right now trying to pull everyone up by their bootstraps.
Whether that ends up being one step, or two, or four data migrations for instance - it is what it has to be. But the smoother and cleaner I can make all this work, all without losing any data integrity along the way, or any more lost data from hardware failures - the better I can plan it out for a Single Master Data Plan i'm hoping the more saved time and mental sanity will result.
