speed bottlenecks - NAS for 1g/2.5g/5g/10g?

[Twice_Shy]

So my first project is still going to be to build two NAS boxes - but they are only connected by 1gig ethernet to start and more of a bit bucket so I don't consider them a speed concern. A Raspberry Pi could probably keep up. One will be used to offload large amounts of video files (the other is my backup) but there's no time bottleneck that i'm rushing on because it starts more a learning hobby and i'm just shooting 1080p at maybe 50mbit to start so files wont get too big.

The future is different however. Once I start shooting 300-2000mbit rates supported by some of the newest cameras the offload time starts to become a bottleneck when multiple terabytes from a video shoot is the norm. So i'm asking opinions and suggestions for what i'd consider the next speed upgrades - to designing a video offload NAS/server to mostly saturate 2.5gig ethernet, then 5gig ethernet, than 10gig ethernet (since I assume cost climbs to the sky over that) i'm going to hit one or more walls. (most obviously going from HD to SSD, but how much cpu is needed to keep up with 5gig and 10gig for instance?)

The offload will happen from USB to the NAS as fast as local connection allows (it could be 1050MB/sec from a samsung T7 drive for instance commonly used for field shooting, 540MB/sec for the T5) after which it would back up the files to the second NAS and then either push files to the workstation in the background to start work later, or if the ethernet is fast enough I just load and edit right off the server. I just want to empty the T5/T7 drives via USB faster so the footage is backed up and there's the option to turn around and go shoot again while someone else starts to preview footage.


Starting on 1gig SAS2 and spinning rust even my box of older drives should work just fine.

2.5gig I assume the newer hard drives should keep up seeing speeds over 200MB/sec when i've looked. SAS2 should still be fine. I'm assuming any cpu would be fine still.

5gig i'm assuming either I need RAID0 for speed or SSD's - but the SAS2 bandwidth still shouldn't be a bottleneck. Is cpu a problem yet?

10gig i'm guessing SSD only because RAID0 stripes get less viable, and I have to upgrade to SAS3 finally. Other servers i've seen built for this have used decent xeon processors but I don't know whats overkill.

 

[i386]

14x 16TB hdds in a hardware raid 6 can read/write ~2.1GByte/s -> for sequential & large io you don't need ssds to saturate 10GBE
If there is nothing else going on but storage then any "modern" (pcie 3.0 capable) cpu should be enough.
Multigig copper is expensive, what about (q)sfp(+/28) ethernet?

Other servers i've seen built for this have used decent xeon processors but I don't know whats overkill.

Storage server use dual cpus for maximum memory (read cache) and high clock for the storage software

 

[Twice_Shy]

So a hardware RAID card could let me use spinning rust if I decide I have to... i'm wondering how software RAID with SAS2/3 connection would work though. How much RAM for cache is a good rule of thumb for saturating 2.5/5/10? I'd think with modern cpu speed "any" cpu should be enough (even a low end celeron) because i'm not needing to do more than pump data - no encryption overhead or transcoding or any of that - unless software RAID is a bottleneck. Just a rapid transfer from USB SSD to internal (SSD or HD stripe) and then pushing copies (with a script) to at least two other computers on the LAN.

 

[jimmy_1969]

My advise is to stay away from software RAID as it typically offers slower performance and poor portability. See here for why. When your controller (stand-alone or integrated on motherboard) dies restoration is a nightmare.

With the risk of stating the obvious, you will not see GByte/s read/write disk throughput unless using any of the higher tier RAID schemes, e g RAID6. Setting up dual NAS each w/ RAID6 means minimum 10 HDDs in total (excluding OS storage). Depending on the volume of data to store this will push your budget beyond what most hobbyists could afford.

You mentions that

there's the option to turn around and go shoot again while someone else starts to preview footage

.

In a semi-professional context, a more cost efficient strategy would be to have one of the NAS servers acting like an ingress node for recorded data, and make sure to validate the transfer after file copy. This ingress NAS should have a minimum redundant storage solution, e g RAID1, preferably more high-and if you have the budget, e g HW RAID6. You probably want to have a 10G LAN for editor workstation(s). Personally I would not put too much emphasis on how long the backup replication between the 1st and 2nd NAS takes.

B R

//Jimmy

 

[Twice_Shy]

Thank you for the useful article link. Reading it carefully I still think I would consider software RAID if the speed could be there - I guess I thought modern cpu's esp with all the cores should be able to keep up, and wasn't sure if RAID5 or RAID6 for video ingress was better than RAID10 or RAID01 if a pair of SSD's (or can you RAID0 3 drives?) was within range of a target speed. I guess it's spend in one place or spend in another - I need fast, safe, and simple to try recover if common failure modes occur. I'd be curious what the overhead is on a typical cpu for different RAID levels and # of drives - seeing the right data might make me say "Oh." and purge the thought of software RAID completely.

Even with hardware RAID6 "RAID is not a backup" is in my mind, wanting a faster 2nd backup on a physically separate server (not just a 2nd drive in the same server) is a peace-of-mind issue. I'm Twice Shy because of past data loss nightmares. "Two is one and one is none", i'd want two copies of data everywhere once shot or I feel naked and wont be happy until the footage is md5'ed and migrated to a pair of LTO tapes! That's separate from wanting the data to be available for reviewing dailies and early editing though - which is why I wanted to saturate ethernet so several people can do that at once like a lower budget jellyfish fryer. (since even 2.5 or 5 is worth doing if I can't do 10)

My first goal for speed of video ingress would be to match the Samsung T7 which is 1050MB/sec via USB 3.2 - CFexpress goes even faster, but those drives are much smaller due to cost. Emptying 2tb of "shot all day" footage would be more commonly experienced than emptying 128gigs of CFexpress "special footage only" for my use I mean cuz i'm trying to minimize storage cost. At that speed, a half hour to offload 2tb? (then another half hour to verify the copy) This is an acceptable turnaround time for me.

I'm curious whether SAS HBA's all have hardware RAID functions (i'd think they would though I don't know how flexible the configuration options are) - if they do my question on software RAID may be moot because i'd plan to stick SAS cards in every server anyway.

 

[i386]

I'm a windows guy and use hardware raid and I disagree with that article and the stuff about software raid ._.
Hardware raid controllers are nothing but small computers with ppc or arm cpus running proprietary software raid...

Thank you for the useful article link. Reading it carefully I still think I would consider software RAID if the speed could be there - I guess I thought modern cpu's esp with all the cores should be able to keep up, and wasn't sure if RAID5 or RAID6 for video ingress was better than RAID10 or RAID01 if a pair of SSD's (or can you RAID0 3 drives?) was within range of a target speed. I guess it's spend in one place or spend in another - I need fast, safe, and simple to try recover if common failure modes occur. I'd be curious what the overhead is on a typical cpu for different RAID levels and # of drives - seeing the right data might make me say "Oh." and purge the thought of software RAID completely.

If a single core arm 1.2GHz cpu can push 2+ GByte/s io, what do you think can a 64bit x86 cpu do with a lot more instructions, much higher clocks and 4+ cores?
Software raid is fast enough on modern cpus (basically every 64bit cpu) and the cpu load is negligible.
Only when you start adding encryption or compression on top of raid it (be it hardware ort software) it will become slower. Or if you use a full rack with nvme device all connected to a single cpu...

 

[acquacow]

My advise is to stay away from software RAID as it typically offers slower performance and poor portability.

I argue the opposite, and I have 20+ years of enterprise experience using mdadm for software raid in production. It's highly portable, works on anything, you can adjust rebuild speeds, configure alerts/automation around it, etc...

I've used it to configure 200TB+ all flash arrays with DRBD on top to mirror to another box for extra protection/etc.
I've pushed it over 250GB/sec as well.

-- Dave

 

[TRACKER]

Absolutely agree with acuacow and i386 regarding software raid.
Back in mid 2000s (2006 i think) we had@work EMC Clariion CX500 and one day service engineer came to "update the firmware or storage controllers" and guess what - i was amazed when that guy connected to internal management console and i saw some windows version (not sure if it was 2000 or 2003 srv) and played 'minesweeper' game

 

[Twice_Shy]

I think you both make good points... my gut instinct tells me cpu should be enough, or at least viable, and entire NAS boxes are nothing more than glorified software raid to run certain modes and such anyway including ZFS. Yet I wonder if the additional parity calculations and maybe there's even timing issues once youre striping 5 drives or things beyond a 2x2 stripe/mirror RAID 1+0 I was half considering. I'd love to see benchmarks showing truly fast examples (i've googled and not seen 10gig saturating setups yet I mean) just to see what is or isn't viable at what cpu - but at least i'm clarifying my questions, i'm planning on running secondhand SAS HBA's anyways, and I would think most of those have some kind of hardware RAID modes on the card. If that comes anyways it simplifies and changes the focus for me - try to see if the SAS controller will already have it.

If anyone has software RAID benchmarks feel free to post in this thread in the future, i'm still curious. I'm just going to investigate hardware RAID more first.
[edit: clarified a few words and meaning]

 

[nabsltd]

Yet I wonder if the additional parity calculations and maybe there's even timing issues once youre striping 5 drives

Modern CPUs can use SIMD instructions to run parity calculations at several gigabytes per second. Until you are trying to saturate 50-100Gbps Ethernet, you won't hit a computing wall.

 

[acquacow]

Absolutely agree with acuacow and i386 regarding software raid.
Back in mid 2000s (2006 i think) we had@work EMC Clariion CX500 and one day service engineer came to "update the firmware or storage controllers" and guess what - i was amazed when that guy connected to internal management console and i saw some windows version (not sure if it was 2000 or 2003 srv) and played 'minesweeper' game

Yup, I used to run a Clariion and Symmetrix SAN years ago... glad those days are over, lol!

 

[BoredSysadmin]

14x 16TB hdds in a hardware raid 6 can read/write ~2.1GByte/s -> for sequential & large io you don't need ssds to saturate 10GBE

Hmm, the only way I could get 2.1GB/s out of 14x 16TB drives is assuming 90% read/10% write. Assuming this is for typical home NAS usage/loads this seems about right. Back then I worked with iXsystems to design truenas capable of sequential 100% write to saturate 10gig, I needed many more hard drives than 14.

 

[Twice_Shy]

Is the bottleneck on writing a RAID controller issue or a physical drive issue? I thought modern drives (that werent shingled) were pretty close to read/write at about the same speed.

Since the most important use of my first faster NAS is to write (to dump/offload external USB SSD video files to a NAS as fast as it can, so it's ready to go get more footage again) it's a big deal I mean.

 

[BoredSysadmin]

Most of the write speed penalty comes from raid5/6 parity calculations, and I'm not familiar with raid controller, which does raid6 for "free," as in zero time.

RAID Performance Calculator - WintelGuy.com

RAID Performance Calculator - evaluates performance of different RAID types and configurations

wintelguy.com

using SSDs for video production NAS isn't a terrible idea, just make sure to avoid consumer models, as their low write durability would kill them very quickly in this scenario.

 

[Twice_Shy]

Well the SSD is for acquiring the original footage - the people who used to record to SD cards are now recording to CFexpress for max speed and those samsung t5/t7 external SSD's for max value/space. Shooting 4k with low compression really makes an SSD plugged into a camera (if supported, which most wont) desirable.

I am undecided what to edit on but that's a workstation problem for the moment - maybe a good use for Optane. I'm just trying to get to where I can shoot and store raw footage for a future project.

 

[mattventura]

There's a few things to unpack here.

Costs:
10GbE can actually be cheaper, or only slightly more expensive than 2.5 or 5g thanks to surplus enterprise hardware. Decent NICs start at $25. $150 or so for a managed switch (CRS305). Use direct attach cables, skip the transceivers entirely (those are where 10gb gets expensive). This may change in the future as 2.5GbE switches come down in price, but right now they're still $100+, and that's for a completely unmanaged switch - the cheapest managed 2.5g switch is a Hasivo for $120. You're right in saying that it gets expensive fast as you go above 10, but you can alleviate some of the price burden by using point-to-point links between individual systems (40GbE cards can be found for ~40 USD). 25GbE is actually more expensive than 40 in terms of secondhand prices. Remember, the main reason for 2.5 and 5 existing in the first place is because of copper cabling limitations - if you can just run a DAC cable, there's not much benefit to using 2.5 or 5 until it becomes cheaper and more ubiquitous.

Now, as for how easy it is to bottleneck: if you spend smarter, not harder, you can bottleneck these quite easily. If you use ZFS, it's very easy to throw in some SSDs as a cache, and SSDs hit network bottlenecks fast. This will become especially relevant as, say, you edit a single project at a time - that project will fill up the MRU and/or MFU cache, and thus access will be pretty fast after the initial loading. Then you've still got the RAM caching on top of that, as well as things like compression (though video is not very compressible, so don't do that here). 10GbE can be saturated by a single SAS3 SSD, two SAS2 SSDs, or 6 HDDs (assuming 200MB/s/drive). 5GbE can do about 3 HDDs, while 2.5GbE will bottleneck on just two HDDs. When you get into NVMe, you're looking at saturating 25GbE with a single Gen3 drive.

 

[Twice_Shy]

Right, I knew costs on 10gig CARDS is way down, the only things I wasn't sure about was the rest of the system - switches, cable lengths (may not be able to have all fast links in close proximity ie a basement run to a few loud NAS boxes), things like that. I wasn't sure about interoperability between 2.5g 5g and 10g other than even a switch supporting 10g wont necessarily support the new slower standards. So i'm just exploring everything and trying to refine planning before buying stuff.

My fast links would be potentially between the fast video ingesting NAS and workstations IF I do the route of trying to edit video over the NAS. I don't know if I need to, and 5g might be almost as good - i'm not pathological about needing full bitrate to premiere in realtime 4k 8k whatever for everything. Working on proxies is not the end of the world. I'm not against using a different workflow to save scarce hardware money to start, but time and efficiency are different.

I don't think I want to use ZFS, i've got a long laundry list of dislikes including hardware overhead despite it's strengths and want to stick with SnapRAID which can also work over BrtFS which gives most of the same strengths as ZFS I care about like stopping silent bit rot. ZFS is probably 'better' but its also alot harder with notably more overhead cost. To use it I feel like i'd have to turn into a sysadmin as well and the learning curve and consequences for doing it wrong are more than I want to risk - so looking for simpler solutions to so I can just focus on shooting video and editing.

This site has TOTALLY sold me on certain things like using SAS, and SnapRAID, and other things still learning that I had zero idea about when every other plan I was using (like lots of external USB drives until I ran out of drive letters and lost data to silent bit rot and finding my backup drives had corrupted data too) absolutely hit a wall and forced me to rethink everything.

I have to be tech-savvy enough to design and build systems which aren't painted into a corner again - and that's what i've been slowly learning and feeling out over time. Sometimes I plan out in 3 or 4 different directions until I figure out what future problem i'll hit then I abandon that direction after researching it. So like right now it's figuring out future speed growth vs workflow... it seems like the option are use direct attached storage/ethernet links, using switches, or just storing the files working on locally on SSD's in each workstation.


Too much info for some but for those interested:
My future needs are at least 2 sometimes 3 people working on video files from the same project, with the ability to grow to maybe 6 workstations max in the future..? Any bigger than that and i'd re-analyze the whole system or maybe be able to pay someone to handle the sysadmin duty. It's going to be usually 2 sometimes 3 for years though at best. Basically i'm designing for a small underfunded startup studio thats a side project for several people (not full time at all, not for years anyways/I still have grad school to go thru) to efficiently work on as they get together and have time - evenings and weekends. Available money needs to go to video equipment and hard drives with minimum overhead, and if small side projects come in to help fund expansion i'm trying to spend very efficiently. Most equipment will start as 'obsolete' (as old as DDR3 era sandy/ivy bridge xeon workstations) but if full time work comes in we need to be able to shift to that and upgrade accordingly without being painted into a corner.

SAS, SnapRAID, BrtFS, D2D2T (disk to disk to tape) migration of data/ backups / accessible offline media archives, I think I have things figured out for just having storage on the minimum cost. That's more than enough for shooting 1080p which is all i'll be doing for the next 2 years or so. I'm just trying to figure out the speed upgrade path for when I can afford 2-3 Panasonic GH6 to shoot 4k to Samsung T7's, that's my upgrade path for needing a solid plan for fast video ingest servers and considering editing directly over shared storage on ethernet. And I want to plan for whatever future new tier will be beyond that too.

 

[CyklonDX]

While not described;

Keep in mind that backplane, as well as connectors and raid/it controllers/expanders have their speed limits too.

Backplanes are typically divided up into 2-3-4 channel groups depending how big it is. In most server chassis there are metal plate dividers between them to show where they are.

ex.

There are timings when signal is refreshed from each couple micro's/milisec per channel, and when they can be read/written to.
Good option often is to run with 2 (or more) raid/it/expander cards if backplane supports it - this way there's less time you have to wait for refresh to happen, and access different channel groups faster, as well as beating the throughput of single pcie controller, and ability to assign the load onto cpu of your choosing (while with single controller you are forced to single cpu).

Typically
top SAS2 backplanes max raw throughput is 6x 6Gbps (most often its just 2x 6Gbps)
top SAS3 backplanes max raw throughut is 8x 12Gbps (most often its just 2x 12Gbps)

and i say typically as there are some backplanes that do more, or do less;
It obviously requries more pcie controllers for more throughput, most often best perf. solution is to have dedicated controller per channel. That adds limitation on your raid setups; and thats where ZFS, and software solutions are actually working best.

In my personal tests on sc 24 bay sas2 backplane with 1x 9211-i
raid0 configuration with just 4 sas disks hgst 8tb 4kn

*using red = 534MB/s
*using blue = 492MB/s
*using yellow = 357MB/s



(each channels has its own queue depth too.)

There are numerous pro's of running software raids' when you add multiple controllers, and lay your disk configuration correctly - giving you ability to enjoy its full potential; biggest benefit is your ability to set cache amount of system ram, or safe flash device like nvme with powerloss protection - cheap ex. micron 3400 nvme | worst part of hardware controllers is limited cache, and the more disks you have the more it starts to choke on continues writes.

Worst case scenario for hardware raid controller:
On R640 with perc h740p controller with 8G cache, and 4 disks 2x old sata ssd 250G for OS, pagefile, temp, 2x 1tb sas ssd's application that does a lot of reads/writes. We had following issue where our os raid1 would eat up all cache, and wait to flush the data, while our raid1 application would be gimped as there's no available cache space - resulting in wait/pause times until data is flushed to disks (could cause bigger latency of 200-400-1000 ms to write data). As by default settings that controller forces data to go to cache first before it can be written. Solution was disabling cache for os disks, and replace them with faster disks.



Additionally most sas2 controllers were not meant for ssd's, and do not support trimm;
(won't allow system to run one either, as exposed raid disk is not seen as ssd.) -> Why it mode, and software raid is better here.



(for network performance, you should also manually optimize - i.e. disable irqloadbalancer, and manually pin threads.)

 

[mattventura]

Hmm, I'm not sure if I completely understand what SnapRAID is actually doing. It seems to not actually be a RAID? I might be missing something, but it looks like you'd still just be getting the throughput of a single drive, along with needing to manually divide files between the disks? If performance is a concern, I don't think SnapRAID does what you'd need it to do.

 

[Twice_Shy]

While not described;

Keep in mind that backplane, as well as connectors and raid/it controllers/expanders have their speed limits too.

Alot for me to unpack here i'll have to reread more carefully to understand everything... but if it helps the more complicated SAS2 drive system i'm currently planning for would be about 16 hard drives, 2-4 SSD's, several opticals and an LTO tape drive. Maybe 24 hard drives if I converged two servers into one use later as an absolute max for temporary use/like mirroring one drive to several others. I'm still trying to understand ports vs lanes and how to best connect things up - expanders, breakout cables, direct attached to the HBA, more than one HBA.

Hmm, I'm not sure if I completely understand what SnapRAID is actually doing. It seems to not actually be a RAID? I might be missing something, but it looks like you'd still just be getting the throughput of a single drive, along with needing to manually divide files between the disks? If performance is a concern, I don't think SnapRAID does what you'd need it to do.

I suppose it's vague if you don't know it... SnapRAID is like a software raid, instead of doing data protection from the bottom up (ie working on data blocks under the file system) it works from the top down (creating parity files within the file system on separate drives) so that if anything is corrupted or lost it can be restored.

SnapRAID isn't used for 'raid performance', it's used for file protection. The idea is you set up something else, like hardware RAID0 or a software RAID0/5/6 system for performance under that, or also in software, or you just have fast drives like SSD's. But it will bottleneck if your parity drive is slower than the drives it is protecting. Since it works on a file system level you can 'layer' it over another performance RAID setup which works on the block level though.