speed bottlenecks - NAS for 1g/2.5g/5g/10g?

[mattventura]

Alot for me to unpack here i'll have to reread more carefully to understand everything... but if it helps the more complicated SAS2 drive system i'm currently planning for would be about 16 hard drives, 2-4 SSD's, several opticals and an LTO tape drive. Maybe 24 hard drives if I converged two servers into one use later as an absolute max for temporary use/like mirroring one drive to several others. I'm still trying to understand ports vs lanes and how to best connect things up - expanders, breakout cables, direct attached to the HBA, more than one HBA.

If the hard drives max out at about 200MB/s, then a single SAS2 lane should be good for about 2.5 of them, or 3 if you're okay with a minor bottleneck. If anything is going to notice bottlenecks the most, it's the SSDs. You may want to consider keeping those in a separate path that goes straight back to an HBA, since they will easily max out that link.

I suppose it's vague if you don't know it... SnapRAID is like a software raid, instead of doing data protection from the bottom up (ie working on data blocks under the file system) it works from the top down (creating parity files within the file system on separate drives) so that if anything is corrupted or lost it can be restored.

SnapRAID isn't used for 'raid performance', it's used for file protection. The idea is you set up something else, like hardware RAID0 or a software RAID0/5/6 system for performance under that, or also in software, or you just have fast drives like SSD's.

It's sounding like if you have to run a separate software or hardware RAID on top of it to get decent performance plus pool the drives, it's not actually saving you a whole lot of complexity, and might just be adding another point of failure. Plus, if you back it with a big array, wouldn't you need your parity drives to be as large as the combined space of the array? You'd have to use yet another array for that. If the extra parity drives aren't contributing to throughput, you're also leaving free performance on the table. If you're backing it with a RAID anyway, my understanding is that you lose SnapRAID's advantages of being able to recover partial data if too many disks fail. Seems like what you actually want is just a simple replication from one array to another?

I don't want to shut down creativity or exploration, but you should be aware that what you're planning to do is most likely going to be more complex and a worse end result than just learning a bit more about ZFS. The thing is, "Overhead" isn't much of an issue here when the alternative is "all my disks that I use as my SnapRAID parity contribute nothing to disk throughput nor capacity". As you said, you want to "design and build systems which aren't painted into a corner again" - but creating a system with this many separate layers instead of using a single vertical solution that does it better and more reliably is exactly how you work yourself into a corner. The problem of storage stacks accumulating more and more layers of cruft over the years is exactly why ZFS exists in the first place.

 

[Twice_Shy]

It's sounding like if you have to run a separate software or hardware RAID on top of it to get decent performance plus pool the drives, it's not actually saving you a whole lot of complexity, and might just be adding another point of failure.

Well for me the learning curve and consequences of doing something wrong with ZFS are too high, and relying exclusively upon a RAID 5/6/10/01 type controller still doesn't solve the write hole problem to my knowledge, so options are limited. I also can't afford an off the shelf NAS that does everything for me, and an off the shelf solution doesn't expand the way I want and need either.

So I want to use SnapRAID on my slow NAS because I need a bitbucket JBOD just a bunch of disks being as simple as possible to manage because i'm not a sysadmin.

Using SnapRAID over anything doesn't really worsen complexity or failure rate - it's designed so that lets say you have data on drives 1, 2, 3, 4, 5. SnapRAID creates parity drives for 6 and 7. The data on drives 1, 2, 3, 4, 5 are untouched, unchanged, if they were RAID 0 or RAID 5 already - there's nothing SnapRAID can do which will take data away. It's comparable to using WinRAR with PARity files, or those usenet PAR/PAR2 programs with all the parity files stored on a physically separate drive. All three things - SnapRAID, WinRAR, PAR work on the filesystem level. It's like taking a snapshot when you run it, and then if data is lost later, you can restore files (or a whole drive) once the parity is calculated. It has more in common with a backup program honestly.


The fast NAS is a separate thing i'm researching. The first use for that is offloading high bitrate raw or prores 4k video from multiple 2tb Samsung T7 SSD shooting drives at 1000MB/sec to a group of internal hard drives possibly in something like RAID 5/6/10 (software or hardware) so the SSD can go right back to work if I need it to/cleared for further work. A RAID 10 or 1 mirroring may be more appropriate to protect from controller failure possible with RAID 5/6 which only protect against individual drive failure.

A separate fast NAS research project is considering using 2-6 workstations to edit a single video project archive. I'm still deciding if I really need to do this, editing with proxies might just be easier. This would have to be SSD to keep up with multiple users though.

Plus, if you back it with a big array, wouldn't you need your parity drives to be as large as the combined space of the array? If the extra parity drives aren't contributing to throughput, you're also leaving free performance on the table. If you're backing it with a RAID anyway, my understanding is that you lose SnapRAID's advantages of being able to recover partial data if too many disks fail. Seems like what you actually want is just a simple replication from one array to another?

The parity drive has to be as large, but not as fast as the combined space of the array. It works kind of like a snapshot or backup, maybe run overnight or during a break.

My throughput ceiling is 1000MB/sec because that's as fast as a T7 can offload via USB 3.2 to it.

What I know of SnapRAID is you decide how many parity disks you want (up to six) and thats how many total drive failures you can tolerate. I don't consider SnapRAID my sole backup despite it working similar to one - it's just to protect the holding area for data in the meanwhile, just like normal RAID arrays. "RAID is not a backup" but i'm saying SnapRAID works similar to a backup program in how it handles things. My data flow is basically Samsung T7 SSD --> two copies of the data with speed and redundancy (so I can go shoot more video) while that transfers --> LTO8 tape. Once I have two copies of LTO8 tape written with my original data, a crash or loss on the fast NAS is not critical cuz I have backups to restore from. The fast NAS is working space to offload and hold things safely possibly emptying several T7 SSD's in one day and writing overnight to LTO8. Really it's like an oversized cache because my goal is to put everything on LTO as soon as possible after which I rest easier.


Learning ZFS isn't off the table I just found it a struggle to understand and the user communities I tried talking to unhelpful and a bit cliqueish. Every post response shouldn't result in "pay me $100,000 per year to design and manage your enterprise grade storage system" or "you're too cheap to really care about your data" and thats all I ever heard on like the freeNAS forums years ago until I just felt chiseled down and stopped asking. I'm not a sysadmin and I don't want to be and I shouldn't have to be, i'm a hobbyist videographer. If something has to be sacrificed in the system i'm exploring it would be running SnapRAID on a file system over other RAID on the block level, but i'm not sure throwing out SnapRAID on those drives actually fixes anything. It's got more in common with a backup program but just solves some of the same problems RAID is meant for. Or maybe I just run SnapRAID over fast m2 SSD's and that solves all speed problems, who knows. This is all an exploration.

Sorry for the length of response but hopefully it's useful to explain where i'm coming from.

 

[mattventura]

Using SnapRAID over anything doesn't really worsen complexity or failure rate - it's designed so that lets say you have data on drives 1, 2, 3, 4, 5. SnapRAID creates parity drives for 6 and 7. The data on drives 1, 2, 3, 4, 5 are untouched, unchanged, if they were RAID 0 or RAID 5 already - there's nothing SnapRAID can do which will take data away. It's comparable to using WinRAR with PARity files, or those usenet PAR/PAR2 programs with all the parity files stored on a physically separate drive. All three things - SnapRAID, WinRAR, PAR work on the filesystem level. It's like taking a snapshot when you run it, and then if data is lost later, you can restore files (or a whole drive) once the parity is calculated. It has more in common with a backup program honestly.


What I know of SnapRAID is you decide how many parity disks you want (up to six) and thats how many total drive failures you can tolerate. I don't consider SnapRAID my backup - it's just to protect the holding area for data in the meanwhile, just like normal RAID arrays. "RAID is not a backup". My data flow is basically Samsung T7 SSD --> two copies of the data with speed and redundancy (so I can go shoot more video) while that transfers --> LTO8 tape. Once I have two copies of LTO8 tape written with my original data, a crash or loss on the fast NAS is not critical cuz I have backups to restore from. The fast NAS is working space to offload and hold things safely possibly emptying several T7 SSD's in one day and writing overnight to LTO8.

So, maybe I'm completely misunderstanding how SnapRAID works, but my impression is that if I wanted to have 6 data disks, and two parity disks, I would need those 6 data disks to behave as single disks. I'd have to manually allocate files between those 6 (or use a layer that sort of does that for me). This means I'd only get single-disk performance on these files. You wouldn't be able to use it in tandem with HW raid, except as a basic replication program, because it can't see the individual disks. Even if it could, it needs files rather than blocks, so it wouldn't work with a SW raid either. At that point, SnapRAID basically just becomes an rsync or zfs-send cron job with extra steps.

My throughput ceiling is 1000MB/sec because that's as fast as a T7 can offload via USB 3.2 to it.

It sounds like this is a pretty sequential workload, so an array of HDDs in whatever RAID solution would probably work just fine.

 

[Twice_Shy]

So, maybe I'm completely misunderstanding how SnapRAID works, but my impression is that if I wanted to have 6 data disks, and two parity disks, I would need those 6 data disks to behave as single disks. I'd have to manually allocate files between those 6 (or use a layer that sort of does that for me). This means I'd only get single-disk performance on these files. You wouldn't be able to use it in tandem with HW raid, except as a basic replication program, because it can't see the individual disks. Even if it could, it needs files rather than blocks, so it wouldn't work with a SW raid either. At that point, SnapRAID basically just becomes an rsync or zfs-send cron job with extra steps.

Thats not wrong, other than I think it accesses by drive volume instead of hardware disks, so if you were to set up some software or hardware RAID 0 stripe that becomes DRIVE-C SnapRAID is just looking at that volume. It's top down, using the file system AS files - it never touches blocks or drive configurations. In a way it's one of it's advantages - it means if the "RAID" fails, every disk is 100% as accessible as before. You can add protection, or remove protection, or change protection levels (from 1 to 2 or 3 parity disks) to EXISTING files on existing drives and it will work just fine.

As commonly used it's single disk performance, it's better for a JBOD bitbucket than a high speed array. It's protection not speed, but protection on the filesystem level.

Using it with files, rather than blocks, as far as I know would work with a software raid? Because a software raid should be working on the physical disk or below the volume level I mean - I haven't worked with them esp under linux but assumed you tell the software raid which physical drives to raid together, and then get some new volume name or way to access it. You'd just point SnapRAID to that then. And since SnapRAID is file based it's no more or less reliable than using those software raid drives with no separate file based protection on them. Adding snapraid, or removing it, or changing protection levels changes nothing on the underlying file system - the only thing it works through on a write level are those parity disks. All 'data drives' are treated as read only UNLESS youre restoring missing files to it. (up to and including the entire volume being gone or the physical disks under it dying and being replaced)

FWIW, some of the other things I find more useful with snapraid that I dont get with ZFS:

- minimal hardware requirements (there used to be a suggested use of 1gig ram per TB of hard drive space, so when I first researched it more than 32gig ram was hard to find and expensive on motherboards limiting your drive array to 32tb)
- easy to upgrade volumes in place and it sees it. I can pull 8tb drives and put in 10tb drives and snapraid wont care, long as that parity drive is also 10tb. No reconfiguring the whole system and remigrating
- easy to add drives whenever I want, i can go from 3 to 5 to 7 data drives and it doesn't care. Or go from 1 to 2 or 3 parity drives. Or drop back down to 2 parity drives. You rerun the program and it just resnapshots from there.
- easy to shrink drives as long as the files fit. I dont have to silver a 10tb drive, have only 7tb of space used but it's impossible to replace with an 8tb drive after.
- easy to run drives right up to the max of space without bogging it down. For something like a media drive that's meant to just be read only it shouldn't matter if i'm using 99% of available space, i've heard ZFS cant do that.
- i've heard of problems with ZFS replacing the same drive of the same size since it cant be even one byte smaller
- god forbid you try to restore data from drives in ZFS if ZFS itself dies...

So I don't have to provision everything at the outset (ie designing a 32tb array of exactly 4x 8tb), i can replace drives, grow drives, shrink drives if data is moved around, add drives, add parity, use the WHOLE drive instead of wasting another 20% on top of other RAID overheads, not lose any data to a proprietary format that you can't access with normal drive tools (also a RAID 0/5/6 problem) if the drive still works, lower power use (i'm not spinning all drives constantly to access one bit of data) which matters when the NAS gets bigger/more like a Massive Array of Idle Disks.

For the slow NAS I was originally looking at how Backblaze did things, I was looking at how Drobo did things where they had the hyper flexibility (pop in drives of any size) but seemed to use proprietary formats on inserted drives which I didn't like, I mostly wanted protection against silent bit rot and drive failure. I think unRAID does something similar too but it's commercial.

You lose automatic drive pooling but there's other software that can do that if that's essential. I'm not 100% set on needing drive pooling - it's convenient from one perspective, but my goal is to migrate all data to offline LTO tape archives of related videos. So instead of having video footage I offload scattered all over a dozen tapes i'd rather try to put like with like and when i'm ready to fill a tape most of the way I just write it. That way if I have to pull the video back off i'm not hunting for which tape it's on. So the 'downsides' arent' so much downsides in my case.

Can SnapRAID's protection work over a software RAID speed solution - i'm not sure but as long as i've got backups and data in at least two places i'm willing to experiment some. I still consider alot of the ZFS filesystem to be experimental, at least to me, because reading thru things of how to configure one makes my head spin. I'd need backups to use that too. Not a deal killer, just explaining why ZFS's strengths don't match my strengths and SnapRAID's weaknesses aren't really an issue to me. However if I make an all SSD fast NAS for editing I may have no choice and may experiment with ZFS then - yet even that will back up to a SnapRAID system mirroring files and changes. My point is I need SnapRAID systems working first no matter what I do. The experimentation comes later.