I think I want to use... SnapRAID! (talk me into or out of it)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
SANs are designed first and foremost to be reliable, performance is second.

To be honest a SAN is not what you are asking for as they get exponentially more expensive as you grow larger and faster.

Both snapraid and zfs can scale as large as you want, the beauty of both is that they are very flexible about hardware upgrades. ZFS is a little more stringent on disk requirements with the parity. If you have large changes to files then you end up at ZFS as the most common choice for roll your own storage.
Well by my Phase 3, reliability and not interrupting work will be paramount. :p Though yes it is true, I am looking for alternatives which accomplish the same goal! That's why I want to work out everything on paper including future scale out plans, so that I dont find myself painted into a corner, realizing my choices are preventing me from taking the next step and not knowing what to do now. (that was what I ended up with when I just kept storing data on external USB drives - only to find my data corrupt later from silent bit rot and having no idea what to do after the damage was done)

I like the fact they can both scale up - that means i'm not forced into a change, but rather can choose when to change. Like if the manual snapshot nature of SnapRAID starts to get annoying or feel burdensome, I might migrate it later to ZFS, only after I have a bunch of matched drives to set up as /vdev pools, striped and mirrored in the right way. SnapRAID is the cheapest growth system that I can upgrade bit by bit, after things are more established I might rush right away into a different system - or I might stick with it for years. But at least I wont be caught flatfooted in another data flood and we have more data coming in from film shoots than we know what to do with, it's just "buy drives and write tapes" on the minimum budget.




I think its not so much that a SAN is required for the highest levels of performanc

The current trend seems to be more towards software-defined scale-out storage, possibly combined with a hypervisor to build a hyper-converged solution. Modern high-end storage is getting so fast that accessing it remotely, even over say a 100-gbit IB fabric, adds very significant latency - remote-access would eliminate the entire point of doing things like putting 3D-Xpoint storage onto the memory bus.

Also don't look at the prices of proper enterprise gear, and immediately eliminate them due to cost.

My recommendation to you, if you want to keep things cheap/control costs, is start building a hardware platform that you will be able to use for years to come, and that you can continue to add to without having to replace things.

Probably start out with something like a used supermicro SC836/846 chassis

Stay away from hardware raid cards that lock you into their specific feature-set - keep all the smarts of the system in software on top of generic hardware.
To the remote access scale out stuff, that was another reason I was curious about FibreChannel and Infiniband, especially with how cheap decade old generation stuff is. (the 4/8gig FC and the 10gig IB) I might not use them for the NAS Data Vault, but they might matter for a highest performance array due to lacking the overhead of TCP/IP and Samba or whatever other filesystem accesses may need to be done. I am not sure where the bottleneck is when doing massive operations on 8k video in Adobe CC - but usually it will be the scratch disk system. And sharing one such system with minimal latency may be more affordable than one per system at times, at least at first - like a single Intel 750 NVME shared to 3 workstations via IB. Ideal would be an Intel 750 per system, but the older IB gear is so cheap it doesn't cost much to let 2 other systems share the main systems SSD. I cant imagine 100gig IB ever being a bottleneck(!), what kind of systems does that occur on??

Being interested in SANs - if a cluster does the job better, that is fine too. I just want redundancy and high availability so that work isn't interrupted at some future point. (this is my 'Phase 3' upgrade after ample storage, and ample performance are attained) I read something about SAS dual porting - if it were something like literally power on either of two servers accessing the same drives i'd be totally content with that. Especially if the SAS expanders themself are easily swap out so that any point of failure has either a backup on the shelf or already set up if you get to work at 9am and there's a problem. If I can set this up with the same software and tools great, if I have to consider migrating to a totally different system - but good to know in advance whats on the list.

Concerning the chassis and stuff - one thing I like about how SnapRAID seems to work is even if I start with something as crappy as my older Core 2 Duo box with some pcie SATA adapters and say 11 drives in an ATX box - I should be able to buy that Supermicro chassis at any other time, swap all the drives over, and nothing is impacted. No data to migrate, just plug in the new LAN cable because the data is already on the drives afterall. I can't afford a 24 bay hotswap chassis tomorrow - I can just afford the drives our video data will be going onto for now. :) (esp after buying the $2000 Ultrium drive) But I can plan to upgrade to that or watch for deals in the future. That sounds like it's less straightforward on a FreeNAS ZFS system, and even if it isn't ZFS wont let me add and upgrade drives piecemeal to the existing storage pool. It was the frowning on my face I noticed when planning how I was going to have to buy matched sets of 8 drives at a time, migrate all the data over to a new server bought with 8 new drives then, just to properly upgrade space that made me look for alternatives to begin with.


What? I have 2x 86TB and 2x 116TB FreeNAS systems... I know I've seen a bunch of people talking about systems with much higher capacity than myself on the freenas forum.

My FreeNAS servers have 32GB of RAM and 64GB of RAM. Definitely not 1:1.

Also, it's common for people to add disks in pairs to ZFS and use mirroring for easy, affordable expansion.
Well I havent looked in years - my massive data loss happened like 5 years back, and nobody wanted to go past 32tb without telling me "hire a professional"/beyond DIY. And everyone was adamant 1TB HD - 1gig RAM was the accepted best practices guide as well. Since I couldn't even find anyone with a 64TB/64gig system to ask I already panned the plan at the time.

Even if I were re-sold on FreeNAS I still have the issues of having to buy matching drivesets, migrate data into/out of /vdevs, no easy way for me to back up to tape from it, etc etc. I may look into it as a secondary high performance NAS option in the future though after the Data Vault is online and operative.
 
Last edited:

TuxDude

Well-Known Member
Sep 17, 2011
616
338
63
Or does one SAS controller just expand and daisy chain while being limited to 12Gbps aggregate?
Basically - yes. Except that for say an 8-channel SAS3 card, that means an aggregate bandwidth of 12*8 = 96Gps of SAS bandwidth to share across however many drives you connect using SAS expanders.

how does an 8 channel controller run 24 drives? Does it all run off 'one' of the channels
Using SAS expanders! Think of it like a SAS switch (but not quite - SAS switches also exist, and are different things - you don't need that). You connect a quad-channel SAS cable (8086 or 8087 cable depending if its internal or external) between the SAS card and the expander, giving you 4x6Gbps or 4x12Gbps of bandwidth to the expander (12G sas is still expensive, 4x6 is lots and you can add a second cable to go 8x6), and then plug drives into the ports on the expander. You can daisy-chain expanders off of other expanders and stick 100 drives on a single 6Gbps SAS link if you really want to, but then you'll not have the bandwidth to push all your drives at the same time. But thats all just math - work out how many drives you want to share per 6G (or 12G) SAS link, how many SAS links and PCIe channels the SAS controller has, and how many PCIe channels you have/want/need on the MB for the throughput you want. Actually physically connecting a large number of drives is the easy part. This is why I recommend using SAS controllers/expanders, even if you only ever plan to stick SATA drives in the system. A SAS controller will run 100 SATA drives in a config like that, but a pile of SATA controllers is just a pain in the ass.

If you want a ton of drives in a small space, check out something like this: SC946ED-R2KJBOD | 4U | Chassis | Products | Super Micro Computer, Inc.
Thats 90 3.5" spinning disks in 4U. And there's nothing stopping you from putting in a SAS card with 4 external ports (thats 16-channel) and plugging 4 of those into a single NAS head. Or stick 4 of those SAS cards into that system, and have 16 of those disk units attached to one NAS. Scaling up direct-attached disks is easy. But for cost-effective use for you, look at used SAS2 JBOD units with built in expanders, either 4U/24-bay or 2U/12-bay units and just use more of them (you can also daisy-chain most of them off a single SAS card if you want). Lots of cheap options in that area.


As for high-speed links - you need to understand the difference between bandwidth and latency. As they say, never underestimate the bandwidth of a station-wagon full of LTO tapes hurtling down the highway. When you're looking at NVMe drives and such, the reason isn't because they can do 2GB/s of throughput in a single drive - its because they can do hundreds of thousands of IO's per second, and for any given read or write command, they return the result virtually instantly. It's like comparing your internet bandwidth to your ping time - who cares if you have gigabit internet speed if your ping is 1000. NVMe has latency low enough that any kind of remote access is going to have a very significant impact - keep your high-speed scratch disks in the local workstations, use the NAS for bulk storage, backups, or sharing files between users.
 
  • Like
Reactions: Twice_Shy
Btw thank you so much for your massive brain dump and indulging all my questions, I very much appreciate it. Esp thinking I would never have considered SAS otherwise.

Using SAS expanders!

As for high-speed links - you need to understand the difference between bandwidth and latency. NVMe has latency low enough that any kind of remote access is going to have a very significant impact - keep your high-speed scratch disks in the local workstations, use the NAS for bulk storage, backups, or sharing files between users.
Okay so I think i'm getting a better handle on this and am pretty well sold on SAS for the high end. Basically the suggestion is to get like a 6Gbps SAS Expander because those will be 'last gen tech'. I have the option to send more than one cable to the expander to up bandwidth - but there's no real limit at least for my needs on numbers of drives. (128 devices being beyond what I can conceive of doing anyways - esp with the 48 device SnapRAID limit and maybe 16 other devices) I dont fully understand this part but dont want to overly complicate small details yet, just need to know it works that way. Using SAS tech I should be able to saturate 10gigE now and 40gigE (or 32gig FC or 40gig IB) just using multiple channels and likely SSD's for the faster speed.

Is there any issue mixing multiple consumer sata, enterprise sata, and SAS drives in the same chassis thru the expanders? Or do I have to start matching sets on any level?


Rolling back to the mid-level system (in spite of the advantages of SAS i'll still have to run the numbers and so am just comparing against an alternative) what about SATA port multipliers, my understanding is that should be the lowest cost overhead per-drive, i'm just wondering what the more affordable controllers are that support that. From what i've read i'd want the FAS ones that can access multiple drives at once. It would be nice if I could saturate 10gigE from a setup with these as well so don't want to stick too many drives on an x1 link I mean. If the financial numbers per head to set up a system like this aren't much better than the SAS option obviously i'll abandon it, i'm just trying to be thorough. I dont consider buying the SATA port expanders a loss because upon server upgrade they'll just get repurposed to other uses in other machines.


Concerning shared SSD (NVMe or otherwise) i'm aware of latency, hence the interest in IB/FC since both were optimized for minimal possible latency - i'm assuming an SSD over IB/FC is still faster than a hard drive locally esp once there's any random access involved. One expected interim system was three machines sharing an Intel 750 1.2tb SSD over IB/FC - the SSD could only be max speed to a single computer used locally which is fine - i'm aware even 10gig IB wont hit the max speed but the 2nd/3rd workstations would not be commonly employed, used when needing to throw more processing cores at problems of separate files, at the same time.

Obviously this is a really specialized tool and in the real world will only be money ahead for some specific use cases - but i'm still curious. Because the alternate version of this plan was a RAMdrive using decade old server parts and the oddball RAM that is insanely cheap per Gig cuz nothing else uses it, letting me throw 256gig or more into a board. That stuff might well saturate 40gig networking links even - but wont be locally wired - Infiniband gives real low latency for a network, as low as possible really, but no lower. I'm wondering at what point a 40gig IB RAMdrive is lower performing than an NVME local SSD due to latency issues.