I think I want to use... SnapRAID! (talk me into or out of it)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
SnapRAID is available for windows as well as linux. I am probably going with linux version so I posted it here, although that's not absolutely set in stone.

Short version of my project: storing lots and lots of digital video (up to 8k and multiple camera motion capture), designing a 32-300TB scaleable NAS to hold and work on that data, digital video data needs to go D2D2T (disk to disk to tape) finally ending up on Ultrium LTO6 tapes.

Things I like about SnapRAID:
- More large systems in existance than systems like FreeNAS (lots of people seem to have 70TB and up, 24 drives a not uncommon configuration, when I looked into freeNAS over 32tb was uncommon and RAM limits under ZFS were an issue)
- Very low system requirements (easy to scale up to those big arrays, even 8gigs seems enough, no "1 to 1" RAM to Terabyte match needed like with FreeNAS ZFS), a 300TB array is doable today if I could afford the drives apparently.
- Lower power use (spools up one drive at a time when needed), was considering a MAID array (not 100% required just contemplating if it's an always on PC)
- No proprietary filesystems (easier possible recover using existing tools if a drive partially dies - no "total pool loss" nightmares like i've heard of with ZFS), I like the fact that it works OVER the file system without changing, you can even use it with data already in place/nothing to migrate in. Or stop using it without having to migrate out data. The separate parity data is just discarded if no longer used.
- Provides robust file integrity verification (my main reason for originally wanting ZFS) checking the whole system with a single command
- Provides protection like RAID does for 1-6 hard drive failures, currently ZFS has nothing beyond triple parity
- Upgrade disks in place at any time. Just first mirror it then swap it. Less annoyance or workaround. Less rebuild time than it sounds like FreeNAS/ZFS will use.
- More easily scaleable. Start with disks of any size. Start with any number. Upgrade size or number whenever. Pretty sure I can downgrade too.

Downsides:
- Doesn't automatically run, I have to schedule or script the snapshots, integrity verification, and similar.
- No built in storage pooling.
- May be limited to single disk performance. (I dont know if anyone has set up RAID under SnapRAID but I dont know why it shouldnt work, though it's not built in. Certainly no state of the art SSD caching and such like i'm told FreeNAS has.)
- May be limited in multiuser scenarios.


So far i'm pretty sold. I can work around the downsides for now, especially since the primary NAS may become a higher performance NAS/SAN in the future, but then the secondary NAS/backup/mirror will still have SnapRAID, and is still used preparing data for backup by the Ultrium LTO drive. So i'm sure SnapRAID is what I need for the second NAS, and if I have to learn it for that I might as well start by using it on the first NAS, even if the performance bottleneck limits me in the future. At least I wont hit limits for data scale out since I need minimum overhead cost per drive and "pay as you go" drive upgrades to store new digital video data. Heavy editing/processing wont happen until notably later probably and can be separately funded.

THAT ALL SAID i'm here to learn and am wondering if there are even any viable alternatives concerning my needs. :)
 

TuxDude

Well-Known Member
Sep 17, 2011
616
338
63
I've been very happy with my snapraid setup since I started using it - you've got the pro's and con's pretty much all listed there.

Yes - you can downgrade disks, in either count (easy to do), or capacity (same as swapping to a bigger one, though I don't know why you would). So it's easy to eg. move from 10x3TB disks to 5x10TB disks - though it will mean a fair bit of IO in recalculating parity, but just a one-time thing.

Performance on snapraid is ... different. Better and worse, depending on use-case. A single file is stored on a single disk, and will get single-disk performance from it. On the other hand, if you have 10 users reading 10 files that are stored on 10 separate disks, then not only are all 10 disks contributing performance, but each disk is also only handling a sequential workload (assuming those large video files you mentioned), which will perform MUCH better than a 10-disk array running a random-IO workload (assuming not SSDs). In my case (a media server streaming to multiple clients) I almost always see better performance now on snapraid than I did when I ran md-raid on the same hardware before.
 
  • Like
Reactions: Twice_Shy
I've been very happy with my snapraid setup since I started using it - you've got the pro's and con's pretty much all listed there.

Yes - you can downgrade disks, in either count (easy to do), or capacity (same as swapping to a bigger one, though I don't know why you would). So it's easy to eg. move from 10x3TB disks to 5x10TB disks - though it will mean a fair bit of IO in recalculating parity, but just a one-time thing.

Performance on snapraid is ... different. Better and worse, depending on use-case. A single file is stored on a single disk, and will get single-disk performance from it. On the other hand, if you have 10 users reading 10 files that are stored on 10 separate disks, then not only are all 10 disks contributing performance, but each disk is also only handling a sequential workload (assuming those large video files you mentioned), which will perform MUCH better than a 10-disk array running a random-IO workload (assuming not SSDs). In my case (a media server streaming to multiple clients) I almost always see better performance now on snapraid than I did when I ran md-raid on the same hardware before.
How long does SnapRAID tend to take to do parity? Does it read all drives at the same time, or does it have to sequentially access drives?

Can you stick RAID 0 pairs under SnapRAID to get more performance?

I was wondering if SnapRAID could work in the future with my theoretical Infiniband/Fibrechannel networking scenarios. That exact scenario - 10 users, reading 10 files, on 10 separate disks, is what we would be trying to do much of the time. The real world probably wont work quite that conveniently, but we will be trying to create a workflow that works around limits in the system. As to why not just have local disks on each workstation instead - the central SnapRAID server makes for easier redundant disks covering them all in one place, combined with some network boot ability to be explored in the future. If a workstation dies or needs upgrading, we just change or plug the hardware into the network and load the relevant boot image, generally paying attention to the 'dedicated' server drive for that workstation. Alternately perhaps the right SSD caching setup or some additional software to manage SSD caching (even with something like the Intel 750 NVMe shared between everything) hopefully could turn more than one user accessing a single disk into much more of a sequential workload.

There may be performance scenarios still benefitting different NAS structures, but I was hoping that SnapRAID might at least be competitive, or scale up decently for awhile, and having some clear marker of when I need to look at something else. (if ever)
 
Stablebit Drivepool + SnapRaid work wonderfully together. I use snapraid-helper - Home to automate SnapRaid tasks.
Could you tell me a bit more about it? I see something about folder replication and storage pooling on there... one thing I had considered as possible is with that say "10 users, 10 separate drives" multitasking was to have the same files that need high availability (like drive boot images which neednt be huge) to be on ALL drives so it scales out to serve max speed to any of them.
 

TuxDude

Well-Known Member
Sep 17, 2011
616
338
63
How long does SnapRAID tend to take to do parity? Does it read all drives at the same time, or does it have to sequentially access drives?
How long it takes depends on how much IO bandwidth your system has, and how much data has changed since the last time you ran a sync to update the parity files. And yes - it will use all of the drives in parallel, so long as they all have data that is required to calculate the parity. So on a brand new array with empty drives, if you add one file, only one drive will be used (plus the number of parity drives it is writing to). If you fill one drive completely up before you start writing to the second drive, still only one drive will be used. But if you spread your data across all of the drives, then at least most of the time, most of the drives will be used. When mixing drive sizes, say some 1TB drives and some 2TB drives - once you are past the 1TB mark on the 2TB drives, only the 2TB drives will be used as you add more files. Basically it comes down to whenever it is using zero's in the parity calculation (empty space on drives, past the end of small drives, etc.), it skips doing useless reads and just uses 0's.

It also only has to deal with changed data - if you add 1GB of data to one drive since the last sync, it probably only has to read 1GB from each drive to calculate new parity, which is all done in parallel. Re-syncing after adding only 1GB only takes a few seconds.

On my system, with around 15 7K SATA drives all connected through a single IBM M1015 SAS card using a SAS expander, I commonly see overall sync throughput around 1200MB/s

Can you stick RAID 0 pairs under SnapRAID to get more performance?
Yes - just make sure in your design that you don't accidentally build somewhere where you are at a higher risk of data-loss than you planned. But people do commonly do that kind of thing to join smaller disks together when using them in an array combined with larger disks.

I was wondering
if SnapRAID could work in the future with my theoretical Infiniband/Fibrechannel networking scenarios.
You won't be able to do block-layer storage on top of snapraid, but you could do file-level, either SMB or NFS. That will be a matter of taking whatever pooling software you use on top of snapraid, and sharing it out.
 

RyC

Active Member
Oct 17, 2013
359
88
28
Could you tell me a bit more about it? I see something about folder replication and storage pooling on there... one thing I had considered as possible is with that say "10 users, 10 separate drives" multitasking was to have the same files that need high availability (like drive boot images which neednt be huge) to be on ALL drives so it scales out to serve max speed to any of them.
Duplication is one feature I haven't tried with DrivePool, since I think it's primary purpose is to duplicate files to hedge against drive failure. In your scenario, I'm not sure it would spread simultaneous access to the file across all of the drives.

I use DrivePool to just combine all drives into one huge logical drive. SnapRaid can still see the drives behind the pool, and works normally.
 
How long it takes depends on how much IO bandwidth your system has, and how much data has changed since the last time you ran a sync to update the parity files

You won't be able to do block-layer storage on top of snapraid, but you could do file-level, either SMB or NFS. That will be a matter of taking whatever pooling software you use on top of snapraid, and sharing it out.
So I should definately plan PCIe SATA adapters then (not a PCI card) to facilitate the parallel reads... i'm wondering if there is some other upstream bandwidth limitation to be concerned of if I get one of those six PCIe slot AMD boards and slap six drive adapters into it though as a cheaper per-drive overhead cost than buying RAID cards. ($300 for 8 drives and $600 for 16 drives like i've seen is more than i'd prefer and that seems the norm) Like should I have full SATA bandwidth to every channel (even with six add on SATA controllers) or is there another upstream bottleneck at the chipset somewhere, capping say a software RAID SSD solution at least, if not even normal hard drives.

Concerning IB/FC I was curious whether it's easy to tunnel normal file systems through there or not as opposed to through gigabit ethernet. Though that may wipe out any benefits of low overhead

Another possibility is trying to set up a SAN of remote drives, treated as local, so SnapRAID runs on it since I assumed the whole purpose of SAN was to appear local. (just linked thru the fibrechannel HBA instead of big local RAID arrays) Which i'd think would also let me later offline the SAN to add drives to that second chassis without shutting down the primary server even though I can't touch any of the higher drives or perform a sync until it's reattached. This might even let me control power use if I had say two SANs attached to one primary SnapRAID server (16 drives per chassis) where I turn the extra drives on and off as necessary, as spare overflow storage.

(as to why do it this way instead of three separate NAS units - is that syncing across 6 parity drives is a better way to deal with failures than having individual stripes of 7 data/1 parity since 2 down drives in any parity set is statistically more likely than 6 down at the same time in a set of 48 i'd assume)

Just trying to decide what the different ways to scale up to 48 drive potential is while minimizing overhead per drive cost and minimizing bottleneck cost. (SnapRAID is designed for up to 42 data and 6 parity for other readers who may not know)
 

TuxDude

Well-Known Member
Sep 17, 2011
616
338
63
With spinning disks, there's plenty of bandwidth available to stick quite a few drives onto a single controller card - a PCIe x8 slot (common for most SAS controllers) has 4GB/s of throughput for v2, or almost 8GB/s on PCIe v3. Under perfect conditions with every drive doing 100% sequential reads that is still lots of spinning rust. You can easily use a single 8-channel SAS controller to run a chassis with 24 hot-swap drives in the front connected through a SAS expander. Then for later expansion you stick a second SAS card in with external ports and plug a JBOD into that - repeat as needed. Even our Compellent storage array at work is built that way, but it has 16-channel SAS HBAs (4x 8087 connectors per card) in the controller nodes, and I think around 80 7K spindles all sitting on a single quad-channel (single 8087 cable) connection for the low-tier right now.

No - you can't use normal filesystems for FC. FC is block-layer, it happens under the filesystem. You could do IPoIB and run SMB or NFS on top of that to do file-layer over IB, but there's probably no reason for it. Assuming your users using this storage are on desktops with regular gigabit ethernet cards in them, just stick a single (or dual-port) 10G adapter in the server and you'll be fine.

Don't worry about complicated or obscure setups involving exotic hardware and such. You'll never be able to push any of that stuff anywhere near its limits using only spinning disks. Bulk storage build on spinning rust is served perfectly fine over standard 10G ethernet out of the server shared to a bunch of 1G clients. And its no problem at all to build a single NAS head-unit and connect even a few hundred disks directly to it using just a few SAS HBAs and some external disk enclosures with embedded SAS expanders - all fully supporting hot-swapping SATA drives around as required. And you still have the options of running snapraid, or FreeNAS, or whatever other software you want on top of such a platform.
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
- More large systems in existance than systems like FreeNAS (lots of people seem to have 70TB and up, 24 drives a not uncommon configuration, when I looked into freeNAS over 32tb was uncommon and RAM limits under ZFS were an issue)
- Very low system requirements (easy to scale up to those big arrays, even 8gigs seems enough, no "1 to 1" RAM to Terabyte match needed like with FreeNAS ZFS), a 300TB array is doable today if I could afford the drives apparently.


Downsides:
- Doesn't automatically run, I have to schedule or script the snapshots, integrity verification, and similar.
- No built in storage pooling.
I'm not completely sold on your assessment of ZFS capabilities/shortcomings:

IE:
Pretty certain ZFS scales to obscene limits, 100's of TB should not even break a sweat w/ a moderate config so I am not sure it's fair to classify ZFS/FreeNAS systems 32TB+ as being uncommon. Memory requirements are reasonable if you don't dip into functionality like de-dup or only use it on specific datasets.

Don't underestimate the value/time saving of automated snapshot/replication. I've said it before and I'll say it again...I've spent FAR too much time managing manual snapshot/replication scripts that I care to admit. :-(

All that being said I am a novice to SnapRAID tech so take whatever I say w/ a grain of salt and not downplaying your strategy at all.
 
Last edited:
On speed to calc parity

How long it takes depends on how much IO bandwidth your system has, and how much data has changed since the last time you ran a sync to update the parity files.

Yes - just make sure in your design that you don't accidentally build somewhere where you are at a higher risk of data-loss than you planned. But people do commonly do that kind of thing to join smaller disks together when using them in an array combined with larger disks.

I was wondering

You won't be able to do block-layer storage on top of snapraid, but you could do file-level, either SMB or NFS. That will be a matter of taking whatever pooling software you use on top of snapraid, and sharing it out.
Well I guess I was mostly wondering if on a system as fast as yours, how much cpu is needed to keep up, especially parity calculating across such a big array? I'm wondering if there is ever a case in the future where with enough fast drives (possibly even RAID or SSD or both) where a slower cpu (even Atoms or older Core 2 Duos) will bottleneck or hold things back.

Concerning the RAID 0 and data loss, I assumed as long as I maintain the same level of parity (1 physical parity drive per 7 physical data drives, so it might be three pairs of RAID 0 drives and 1 parity for instance) I should be safe. I even thought I might leave parity drives direct connected instead of RAID, it would be slower to calculate/restore parity than across RAID stripes but it could get performance up.

"I was wondering" sounded like an incompleted thought. :)

As far as block layer, my idea was to do a SAN with block level storage under SnapRAID. My understanding was that the SAN should present itself to the server as local storage, and I hoped SnapRAID could simply install over the top of that. But since I don't have experiences with either or there might be some unique error issue somewhere that SnapRAID doesn't like.


As i'm trying to refine both my language and the idea in my head of "what I need and why" I guess i'll insert this clarification - I posted asking on "Enterprise Storage" but talked of constrained budgets which are generally contrary objectives. But in many cases "saving money now" isn't a plan to run cheap forever - it's more about postponing costs until absolutely necessary. For instance if we have an opportunity to use the Red Weapon this weekend, and we can scare up a few hundred dollars between me and the actors, were buying another hard drive or two. If we have empty space and dont need a drive and can scare up the money, maybe it's time we finally do that motherboard upgrade from "the old core 2 duo beater box" to a new Supermicro server board made to be up 24/7. Or go from a couple cheap SATA cards to a proper RAID controller to boost performance with existing drives.

Looking around things like SAS attached chassis working with normal SATA drives, browse some and I say wow this stuff is nice but then I see that it's 3 grand without any drives for instance for a 16 bay rackmount and I remember why i'm originally trying to do some things on the cheaper. It's nice to know what represents the high water mark (so that I know what i'm comparing against) as well as the low water mark (the minimum cost of entry, cobbled together on a shoestring) so that I can prioritize upgrades - mostly based by what improves production workflow or reliability for instance.

As a random example, I was reading around this lately: SAS vs. SATA - EnterpriseStorageForum.com but I can't tell whether it's Fear Uncertainty Doubt based marketing about error rates 10,000 times higher on SATA, or whether proper programming gets around it. I've always thought it strange that many hardware RAID arrays if they hit one bat bit apparently kill out the drive ignoring the rest of it - one bad bit in one bad file only affects one file (or ideally just that bit) - I see no reason it couldn't pull a good bit from parity elsewhere, then still use the rest of the existing working files in case the parity drive had one bad bit later in the drive for instance. Or even worse silliness how RAID controllers have to read an entire drive even if it's not full, like could have 1 gig of data on a 1 tera drive and if I hit three single bit errors anywhere on a RAID 6 array of them I guess I lose all of my 1 gig data even if the zeroed bits are in the free space. (?!?) I'm hoping SnapRAID operates more logically though.

A semi related topic concerning damaged disk recovery (you are such an encyclopedia of other info I guess i'm sticking this in) - I wonder if there are any programs that will restore as much as possible of a damaged file from disk, instead of returning an error (and leaving an incomplete file) just reading as much as it can, even possibly from both sides (stepping back from the end as well as forward from the beginning, or even jumping into the middle of it to try and get past a read error) even if it zeroes out the remaining data and is obviously going to have corrupt parts. Because a large file with even a few corrupt bits would still able to calculate some parity at least. As the files get larger, and it's as little as one bit flipped somewhere, it should seem possible to reconstruct a file from say two damaged copies that have bit errors in different places.
 
I'm not completely sold on your assessment of ZFS capabilities/shortcomings:

IE:
Pretty certain ZFS scales to obscene limits, 100's of TB should not even break a sweat w/ a moderate config so I am not sure it's fair to classify ZFS/FreeNAS systems 32TB+ as being uncommon. Memory requirements are reasonable if you don't dip into functionality like de-dup or only use it on specific datasets.

Don't underestimate the value/time saving of automated snapshot/replication. I've said it before and I'll say it again...I've spent FAR too much time managing manual snapshot/replication scripts that I care to admit. :-(

All that being said I am a novice to SnapRAID tech so take whatever I say w/ a grain of salt and not downplaying your strategy at all.
Oh I know ZFS is capable of getting absolutely huge theoretically - zettabytes afterall. The problem is that there are no practical and affordable implementations of it. Asking on the FreeNAS boards basically showed me that people hat 32TB arrays, but barely any larger. Even 64TB was unknown territory despite all the people using it. It's like there's a rift between the SOHO desktop level and the Big Iron level of ZFS implementation. Finding boards that had 1gig ram per 1TB ram once you get much over 32tb got more and more expensive, requiring server boards, and lots of expensive RAM, and since it was more experimental in larger sizes I felt there was more risk of a total array loss the way others have had with ZFS even if due to a configuration error or similar mistake.

Even trying to get people to speculate about over 32TB systems on the freeNAS boards was mostly an exercise in frustration, with people giving me the same "hire a professional and start with at least $15,000 for hardware" type answers that were completely useless to me.

I'm also pretty sure SnapRAID will not be the only NAS I ever use, just the first one. I need it anyways because of trying to use it to create parity file which can be written to LTO Ultrium tape, giving me RAIT (redundant array of inexpensive tape) with parity tape reconstruction options. It also scales up super easy with mismatched drive sizes, upgrading individual drive sizes at any time, letting you fill 100% of drives instead of a 20% OS holdback for migration,, not interfering with data recovery of individual drives, no migrating data into/out of a proprietary format (like all hardware RAID and software ZFS both need), etc etc i've gone over it earlier in the thread. The strengths all outweigh the weaknesses.

At some point I expect potential hassles of my cheaper manual workarounds will build up - but nothing prevents me from using a different NAS solution by then. Even something semi pro like a Synology 12 bay chassis or something which my sysadmin friend recommended. I'm hoping by the time I need such a thing we'll have postponed needing to buy it for several years already (while having used the saved money getting our footage) and have 100TB of footage on tape. Or maybe FreeNAS will have more 64TB or even 96TB systems by that point, and building such a set for high performance will be much more normal by then.
 

markarr

Active Member
Oct 31, 2013
421
122
43
As far as block layer, my idea was to do a SAN with block level storage under SnapRAID. My understanding was that the SAN should present itself to the server as local storage, and I hoped SnapRAID could simply install over the top of that. But since I don't have experiences with either or there might be some unique error issue somewhere that SnapRAID doesn't like.
I think you are confusing what a SAN is for. Yes a SAN presents a luns that you mount as disks, but the whole point of the SAN is provide the redundancy you are wanting with SnapRAID. We have a two node san, each node has two controllers, you could literally destroy an entire node and not a single thing would know. Its not that it wouldn't work is that you would be losing space due to two layers of parity. It would be more beneficial to build a NAS with either ZFS or SnapRAID, and use normal shares from there.
 
With spinning disks, there's plenty of bandwidth available to stick quite a few drives onto a single controller card

No - you can't use normal filesystems for FC. FC is block-layer, it happens under the filesystem. You could do IPoIB and run SMB or NFS on top of that to do file-layer over IB, but there's probably no reason for it. Assuming your users using this storage are on desktops with regular gigabit ethernet cards in them, just stick a single (or dual-port) 10G adapter in the server and you'll be fine.

Don't worry about complicated or obscure setups involving exotic hardware and such.
I suppose that makes sense... I was thinking about internal speed potential to calculate parity even faster, but I suppose if you can saturate 10gig Ethernet/Infiniband/8gig FibreChannel from a 12gig SAS controller card that should be fast enough. I'm trying to spec out several levels of potential system here since what starts as being happy if I can saturate 1gig ethernet will eventually need to be faster. I'm wondering if my desire to use IB/FC is silly or not, whether 10gigE HBA's have come down enough or/and the hassle difference isn't worth it. :) I like the idea of multiport IB/FC cards being able to direct plug and have no latency for playing around with cluster stuff, but if the main goal is storage virtualization maybe i'm creating problems for myself.

The main exotic hardware i'd had other thoughts about were along the line of sticking an Intel 750 SSD on a main server, and using the fast no latency IB/FC links to basically share it between multiple workstations in the future to be everyones scratch space. Or something similar involving an older server board and a huge ramdisk since some of the obscure server-only RAM gets ultra cheap per gig when it's a few generations old. Wasn't sure if that would work as well over 10gig Ethernet if treating things as files.

One big thing I like about SnapRAID is that I should be able to start on the cheap hardware I have laying around, and later upgrade to better hardware without affecting the data or parity on the drives. Just build the new upgraded PC and migrate the drives over. Same way as I periodically upgrade individual drive sizes. Being able to postpone a cost of a few hundred to $1000 or more a bit longer is always useful - whenever spare money becomes available in the future there WILL be upgrades, and i'd like to map out what that upgrade path will be, even if it's a year or two off before I can apply it. So learning about SAS and the rest of it is still useful - even if i'm not sure if i'll ever use more than consumer SATA drives with it. (for performance upgrades i'd assume SSD is all I need anyways)
 
I think you are confusing what a SAN is for. Yes a SAN presents a luns that you mount as disks, but the whole point of the SAN is provide the redundancy you are wanting with SnapRAID. We have a two node san, each node has two controllers, you could literally destroy an entire node and not a single thing would know. Its not that it wouldn't work is that you would be losing space due to two layers of parity. It would be more beneficial to build a NAS with either ZFS or SnapRAID, and use normal shares from there.
This is possible - my understandings of SANs were more limited, but it sounded like the highest levels of performance, failover, and redundancy all required it. Part of my future migration path was about upgrading when it could be afforded, to lower maintenance, higher uptime, higher performing systems. So the idea I had in my head (perhaps half baked) was to make a couple of lower budget SAN units that for now are just treated like fancy drive expansion bays. But later would be a mirrored set to guarantee no downtime, possibly under a different NAS/SAN software entirely like I understand Openfiler might have the ability to do. Something where a cluster of 8 drives or whatever is just presented as a single fibrechannel connection and if one crapped out in the middle of a working day, a cold spare would be fired up and plugged in where assumedly it self auto-remirrors back to it, until there's time to mess with the dead chassis to figure out what crapped out in it.

Everything is about 1) start at minimum budget, 2) scale up size first, when that's met 3) scale up performance since thats usually the next bottleneck, and then 4) scale up the uptime/reliability and reduce maintenance overhead by throwing money at the problem when were too busy creating content and the budget is no longer struggling. Hence eventually SAN nodes. But I could have the wrong understanding in my head too - feel free to present alternative plans for what i'm trying to do. I don't want to be in an echo chamber/i'm looking for divergent plans or suggestions for achieving my stated goals/priorities.
 

RyC

Active Member
Oct 17, 2013
359
88
28
I'm sorry if I've missed this, and I know you've talked about wanting to store (static?) boot images, but what general type of files are you looking to store? SnapRaid (and snapshot RAID in general) is not suited for files that change often.
 

markarr

Active Member
Oct 31, 2013
421
122
43
SANs are designed first and foremost to be reliable, performance is second. To get to the performance and space that you are wanting is $$ and you add complexity and management overhead that is unneeded. Practically speaking the difference between SAN and NAS are the exports, SAN block, NAS file, that's it. They both do all of the pooling, export, and raid on the backend, to make them fast they need the same things that have been talked about in this thread multiple times. We have a san from a large vendor and for 20tb of ssd they want an astronomical amount of money compared to what you have said you want to spend on this.

To be honest a SAN is not what you are asking for as they get exponentially more expensive as you grow larger and faster.

Both snapraid and zfs can scale as large as you want, the beauty of both is that they are very flexible about hardware upgrades. ZFS is a little more stringent on disk requirements with the parity. If you have large changes to files then you end up at ZFS as the most common choice for roll your own storage.
 

TuxDude

Well-Known Member
Sep 17, 2011
616
338
63
I think its not so much that a SAN is required for the highest levels of performance - but more that people needing the highest levels of shared-storage performance all use SANs to accomplish that, or at least used to. The current trend seems to be more towards software-defined scale-out storage, possibly combined with a hypervisor to build a hyper-converged solution. Modern high-end storage is getting so fast that accessing it remotely, even over say a 100-gbit IB fabric, adds very significant latency - remote-access would eliminate the entire point of doing things like putting 3D-Xpoint storage onto the memory bus.

Also don't look at the prices of proper enterprise gear, and immediately eliminate them due to cost. Yes, new prices are stupid-high, so don't pay them. There is a large market of used IT gear out there that is quite reasonable and still provides plenty of performance. I've got a 4U supermicro case with 24 hotswap bays and a SAS2 expander backplane in my basement, and many other users around here have similar configs. Fairly recently I picked up an extra SAS2 HBA (non-RAID) for connecting some external disk shelves for around $50 - thats 16 channels of SAS2 at 6gbit each aka 96gbit of SAS bandwith to connect to drives/expanders - far more than the PCIe 2 x8 slot it plugs into. Watch the great-deals and for-sale forums here and have some patience, and you can get some very scaleable gear for very reasonable prices.

My recommendation to you, if you want to keep things cheap/control costs, is start building a hardware platform that you will be able to use for years to come, and that you can continue to add to without having to replace things. Probably start out with something like a used supermicro SC836/846 chassis (thats a 3U or 4U 16 or 24 bay storage-server chassis), with a Xeon E5 V1 generation system in it. You're not going to need a ton of CPU power (I don't push the CPUs in my box hard even during a large snapraid sync, and I'm on even older L5520's), but going to a dual-socket MB will give you a ton more PCIe lanes of IO bandwidth to grow into in the future, whether you end up sticking SAS controllers, 10G NICs, or FC/IB or whatever into them. As a base platform for bulk storage that will easily see you into the few-hundred disks range with plenty of compute and bandwidth to handle spinning disks. If you need high-performance/low-latency it can be combined with local SSDs in each client as working space as money permits. And that box will give you the flexibility to run snapraid, or switch over to whatever other platform you like in the future. Add a second HBA to connect to an external 2U storage box with 24 2.5" bays and you can easily add a bunch of SSDs to build out tiered storage, or for ZFS cache/slog, or whatever. Stay away from hardware raid cards that lock you into their specific feature-set - keep all the smarts of the system in software on top of generic hardware.
 

mattr

Member
Aug 1, 2013
120
11
18
Asking on the FreeNAS boards basically showed me that people hat 32TB arrays, but barely any larger. Even 64TB was unknown territory despite all the people using it.
What? I have 2x 86TB and 2x 116TB FreeNAS systems... I know I've seen a bunch of people talking about systems with much higher capacity than myself on the freenas forum.

My FreeNAS servers have 32GB of RAM and 64GB of RAM. Definitely not 1:1.

Also, it's common for people to add disks in pairs to ZFS and use mirroring for easy, affordable expansion.
 
You can easily use a single 8-channel SAS controller to run a chassis with 24 hot-swap drives in the front connected through a SAS expander. Then for later expansion you stick a second SAS card in with external ports and plug a JBOD into that - repeat as needed.
I just realized I had a bit of a brain lock here, how does an 8 channel controller run 24 drives? Does it all run off 'one' of the channels (meaning I can connect 7 more like that) or what really?

Since i'm exploring the options for future scaleout, i'll just go to a most extreme system I can think of if you want to comment on it:
- 48 drives used for the NAS array (42 data 6 parity), this is not ALL the drives in the system, just the drives served over the NAS
- Hot spare drives (indeterminate number, but at least several)
- SAS connected Ultrium tape drive
- Four consumer SATA DVD/Bluray burners/readers (used for distributing data, like a sample cut of footage like dailies/rushes. Anything more than this we'd have a separate replicator, but for the intermittent use it will see I was just going to throw four in the main server chassis and call it done. Also some people like to send incoming data this way on a spool instead of on USB drives or hard drives.
- Two boot drives in RAID 1
- A random mix of consumer SATA, enterprise SATA, and SAS drives in the array as things get slowly upgraded as needed
- Able to saturate 10gig ethernet consistently
- External drive pluggability, if someone mails a 5TB external or something and you just want to copy it all in. Most of these will be USB3 or Firewire but leaving a spare external SATA, external SAS and similar connectors around make for convenience so we don't have to load it into a drive tray.


What kind of controllers should I be looking at for running this monster setup?? :p Or does one SAS controller just expand and daisy chain while being limited to 12Gbps aggregate?

This is literally the most I can even comprehend of but i'll just set a high water mark to start. Such a system could store half a petabyte now and even more with drives out within 3 years. Ideally things would be set up so that everything was redundant capable (ideally with automatic failover but at least allowing a manual swap) so an entire second server could just be plugged in with any problem on the main, to the existing racks of data, and resume running without interrupting any work.

Could even more performance be needed in the future, yes, but i've already planned separate "working NAS/SAN" options for that. This is more for a master data vault of working data. (which itself is still a subset of the even larger LTO Ultrium tape library) The size of this setup is entirely about how many people are working on projects of what size in parallel, otherwise old stuff can be migrated to LTO then deleted.

There may be better options than SnapRAID when scaling up to this size - what i'm seeking though is ideally to know that SnapRAID could handle this, while i'm learning some other software, before migrating over to that software. So the hardware is totally reused with no wasted money, and i'm free to use SnapRAID until it feels like a chain around my neck or a burden to use, without buying brand new hardware and migrating data all over the place. :)

Again my desired progression:
1) Inexpensive space (until there is 'enough')
2) Performance (until we have 'enough', start with 1gigE saturation, later 10gigE saturation, beyond that TBD)
3) High availability/redundancy (when work interruptions cant be tolerated, to consider clustering options, SAN, any other offered alternative)
4) Max convenience/lowest maintenance overhead (software or otherwise) - just let me buy drives and new hardware if something fails, and not have to do anything else.


I'm sorry if I've missed this, and I know you've talked about wanting to store (static?) boot images, but what general type of files are you looking to store? SnapRaid (and snapshot RAID in general) is not suited for files that change often.
Massive media archives of files that dont change often. :p The idea was to pull up footage on demand, migrate it to something faster (like SSD or a performance NAS/SAN) processing the hell out of it, then stick it back on the server at the end of the day. I don't know if once per day is considered too often to save, or if daily snapshots are going to grow out of proportion, but if it is I can look into hybrid solutions.

Other data like millions of WAV files and stuff for the audio editing/foley setups would also be mirrored - but they wont change often either. Just be added to over time as the collection grows.

I'm pretty sure deduplication and other NAS systems wont help that much either - if were going to be saving multiple versions of files, the data changes (like a different color grading pass, or a different VFX rendering) are probably different enough there is not much to be saved.