Advice building RAID array for data NAS

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

DrStein99

If it does not exist ? I am probably building it.
Feb 3, 2018
115
4
18
50
New Jersey, USA
I am shopping for an array of SAS drives to build a NAS or SAN on my HP Prolaiant 360 appliance with 8 bays. I am using 750gb of data now (on a cheap tower computer I threw together that was temporary - 2 years ago). I think 2TB of data should be more than what I need for a year or two (including copying over my existing data, will give me about 1tb extra to use up). I have 8 bays in my server to use, I thought raid 5 would be good choice, with 1 hot swap. I haven't figured out yet what to do with the BOOT on this server yet, since that partition and it's operating system should probably be on it's own drive to make it easier to manage the data drives, but I lose 1 bay.

So, 7 bays for data, 1 for hot-swap standby, gives me 6 drives of data in Raid 5. 1 whole unit is sacrificed to parity storage, so that leaves me with storage capacity total of 5 drives (if my math is correct). (5) - 500 gb SAS drives gives 2.5tb storage. I was going to buy used drives, probably 10k SAS, since they could probably have a longer life expectancy -vs- 15k SAS. I am only running 1gb network, so without 10gb switches and NIC's everywhere, running super speedy hard drives is just overkill, unless it's just about or equal to the same price.

Can anyone with more experience than me (which is probably alot of people) verify my math here - and give any advice on a better plan and / or save cost ? I am cheap, this is for myself so I do not have corporate unlimited spending cash.
 

rune-san

Member
Feb 7, 2014
81
18
8
The real question is what do you need to do with it? If your needs aren't great and you are cheap, why not just get 2 USB Externals? If you don't need immediate access to your Backup Data in a loss, do you need RAID? Why not just have a disk, then a backup disk? If your main disk fails, just use your backup.

10K Disks and RAID 5 all seems unnecessary if you're just looking for an idle NAS for a couple TB of Data.
 

DrStein99

If it does not exist ? I am probably building it.
Feb 3, 2018
115
4
18
50
New Jersey, USA
I plan to setup and test the rest of my servers to boot and run over ISCSI, or other protocol. I'm running mysql database. Also setting up media server with an array of IP security cameras. Virtual machines, snap images, etc... It would also take forever for me to re-do laptop hard drives, rescue files if I had to wait on the speed of USB Sata.
 

Evan

Well-Known Member
Jan 6, 2016
3,346
598
113
2 x 2tb SSD in raid 1, that’s what I would do if I wanted performance in this case.
However I would want to know what generation dl360 your using, only consider it if it’s new, if it’s older the power heat and noise is not worth it just to provide some storage, just put he storage direct in the systems that need it is more efficient.
 
  • Like
Reactions: fractal

DrStein99

If it does not exist ? I am probably building it.
Feb 3, 2018
115
4
18
50
New Jersey, USA
I will be using a DL360 GEN 7. The noise of this server combines with the 8 existing, and more to come - not an issue. I already have to have special HVAC just for the room.
 

fractal

Active Member
Jun 7, 2016
309
69
28
33
I agree with Evan. 10 pairs as you need capacity and can afford it. USB backup of critical data. A 4TB USB drive will last you years if you only need 2TB now. 2TB SSD's are expensive but 1.2's aren't bad. The good news with 10 is each pair does not need to be the same size as the previous. I would NOT go with 10k spinners any more.
 

rune-san

Member
Feb 7, 2014
81
18
8
I would split up your fast / slow. Find how much space you could reasonably expect to use for Virtualization / Snaps / etc and get enough RAID 1 All-Flash Capacity for it. Put your slow stuff on a big drive (4TB or higher). Backup to a big external USB drive (you can do 8TB for $150). Call it a day from there.
 

ecosse

Active Member
Jul 2, 2013
463
111
43
When you say "other servers", what do you mean - other physicals or just VM's? What exactly is your budget? The budget really defines your options.

FYIW I would probably go with most of what rune-san says - if you *really* need the speed then use SSD's (or if you've the cash) but otherwise I'd use spinners to save money. I hate restoring stuff so I'd use a RAID-6 and not have a hot spare personally. I'd have a separate backup drive as well. RAID is there for availability - if you can afford to restore from backup in the event of a failure don't bother? Every option depends on £.

As far as I can see the DL380 G7 has only USB 2.0 so a USB option is going to be dog slow unless you can find the HP card. For backups its probably fine or file storage if you went down that route. You could use something like stablebit clouddrive and use google drive as your backup repository. Never used this personally but I plan to.

SSD's do provide nice fringe benefits - less noise and heat.
 

DrStein99

If it does not exist ? I am probably building it.
Feb 3, 2018
115
4
18
50
New Jersey, USA
By other servers, I do mean real physical linux servers, 2u, 5u, 1u, and various appliances etc... One of the physical machines host a virtual server, that has many virtual machines.

I am not going to trust my entire collection of designs, photo's, video's on 1 stand alone hard drive connected to a server. Sorry, I can not afford to wake up one day and find this one and only drive has decided to go away. In a data center, I would immediately write-up and put on probation any technician who I found was using an $100 external hard drive to store company data for processing that was attached to a new $10,000 server. I would also demand they take a narcotics blood and urine exam.

Archived data is stored on mechanical drives, packed in fireproof boxes or in a safety deposit storage at a bank.

I use SSD's for my local workstations to process work locally. That work is backed up to my file server, where my large volumes of mechanical drives are housed. (2) raid ZERO striped SSD's, double the speed - not redundant, since their data is backed up to the file server, speed is important on the the workstations that must use Windows.

SSD drives are way too new. There are only a million different makers and models, and I do not have a decade 10 years of experience with any of them enough to know how reliable and long they will last. Many SSD drives are actually slower than mechanical hard drives either writing, reading or BOTH. It takes me days of research to review the data sheets for every one to sort through the advertising and find the fine print. What I can say for certain, if a maker does not publish the actual accurate speed benchmarks openly with a life expectancy on it's front page, and they are going to make me dig for information, than it is only because they are intentionally hiding it because the damned drive is actually slower than normal retail grade mechanical drive.

As far as I can see the DL380 G7 has only USB 2.0 so a USB option is going to be dog slow unless you can find the HP card. For backups its probably fine or file storage if you went down that route. You could use something like stablebit clouddrive and use google drive as your backup repository. Never used this personally but I plan to.
I have an 8 drive SAS hot-swappable backplane with 1gb of CACHE. Even if I had USB 3.0, it would still be at least half as fast as the RAID array. If I had only one workstation, I would still simply install that hard drive to one of the 6 or 8 internal SATA connections. Once 5 or 10 machines start streaming data from that server over 10-gigabit (or even 1gb) connections, that USB serial - data hard drive will just give up! What actual advantage are you trying to explain that a 2-wire USB external drive could possibly have, over the internal hot-swap backplane structure?
 
Last edited:

nk215

Active Member
Oct 6, 2015
412
143
43
49
what's your budget? From the first post, I have an impression that you are on a tight budget. From the last post, money doesn't seem to be an issue.

Mid price range: get yourself a ZFS setup with a Optane drive as SLOG and call it a day (2x 4TB HDD + Optane).

Money is not an issue: 2x 2TB SSD. Lot of options are available at this level.

Or do a hybrid setup: a 4TB HHD served as a backup to an SSD array. I have this setup due to the fact that I have a bunch of SSD of various size collecting dust. I use btrfs to get all those drives into an all flash setup and back that array up to a RAID6 setup. BTRFS let me remove, add drives whenever I want.

I would not get a bunch of small 10k spinners. It just doesn't make sense in term of heat/power usage for a NAS.
 

DrStein99

If it does not exist ? I am probably building it.
Feb 3, 2018
115
4
18
50
New Jersey, USA
what's your budget? From the first post, I have an impression that you are on a tight budget. From the last post, money doesn't seem to be an issue.
I see your point here. I said I was CHEAP, and you are offering cheap options - and I agree, an external USB is the cheapest and least difficult. I have failed to declare what my version of cheap is, and at this point - I confused my own self - I apologize.

I like the sound of BTRFS, I do not know what that is yet - I will read up on it. If I read correctly, looks like SLOG is another layer of cache, on top of raid controller's cache and / or computer memory cache, that will store more data to eventually be written back to the storage array.

With all the options, I am losing focus. I know I need a large reliable storage bank for photography, video, and workstation hard-drive back-ups, and slow-access data libraries. I will not be processing video directly to this storage, it would be for storing pre & post edited video and data that is processed at my workstation level on the local workstation SSDs. After I build this to replace my existing FreeNas appliance (that could be in jeopardy of failing soon since I chose RAID-0 2 years ago when I made it), I will then concentrate on how to perform incremental change to an off-site cloud or co-location backup. I want to make this storage possible for me to add, extend, remove, hot-swap volumes with safe redundancy to failure on any of the physical devices I use to store with.

It is because of the size of 4tb, I get scared scanning for prices on eBay. I know I want to use my SAS 2.5" array to make this work, and am confused by SAS, SAS 2, SATA - SAS, or however many types of drives that can and can not work with each other. Is the SAS array any more reasonably faster than a SATA array? Sometimes these things are measured in milliseconds that would make copying a large file save me 30 seconds on 15 minutes SAS vs SATA array and cost me twice as much - I consider that unreasonable advantage.

Next most important for me, I will probably create a second separate storage bank for fast access to database, virtual machine / iscsi testing. I will be also testing video storage from IP camera's, live video feeds to LAN and WAN which would be run on the faster 10gb LAN network (and 1gb for each IP camera / audio / sip phone).
 

rootgremlin

Member
Jun 9, 2016
42
14
8
i get the overall feeling that you heard some numbers and are now trying to make the most sense of it.

for hard drives and SSDs there are these two factors: Throuput and Latency.

First Throuput: nearly never really matters besides copying large amount of big Files from a to b. It taps out at about 200MB/s for Hard Drives and about 4GB/s for NVMe SSDs. The currently Fastest Harddisks can not saturate SATA2 interfacespeeds (==300MB/s)
For SSDs most of the time the specs of the Interface is the limiting factor. (as in 4GB/s are 4 PCIe 3.0 Lanes of 1GB/s each)

Second Latency: the fastest Harddisks (==15k RPM) have a latency of about 4ms (mili-seconds)
the fastest SSDs are at about 20 micro-seconds (intel Optane), so 200.000 x that fast.

This ALWAYS matters, as it reflects the overall "snappiness" of the System. Also if you copy 1Million 4K files the "slowest" ssd runs circles around the fastest Harddisk.

The best of both Worlds is RAM, something around 5ns (nanoseconds) latency and at best 350GB/s but very NOT affordable in appropriate sizes for storage purposes.

So first decide what you need/want. For videofiles there is no benefit in fast latency, every hard disk is suitable.
For VM/iSCSi/Database usage the Throuput nearly does not matter, but the latency certainly does, so go SSD storage and Price wise, go with just enough of what you can afford.

Now comes the decision if you need the UPTIME, so you would have to RAID your harddrives/SSDs or if you can life with "only one" disk and the fact that your server is going down until you can recover from Backup

Backup is something you schould always do as it is the only thing that can save you from a virusinfection, Cryptotrojan or a "bad admincommand" (rm -rf *)
 

Evan

Well-Known Member
Jan 6, 2016
3,346
598
113
Disks are really good for pure sequential read or writes, they just are horrible for mixed or random workload of any type.

For big disks I would plan about 100 IOPS, 15k disks say 150 IOPS.
But then you have cache’s etc to factored in, I have some 180 disk arrays that run 15-25000 IOPS all day every day, capable of more but the latency does start to go up. This is a very random workload, your workload should be much more sequential but still for the performance you need 20k won’t make it I am afraid. Have to give up something.
 

DrStein99

If it does not exist ? I am probably building it.
Feb 3, 2018
115
4
18
50
New Jersey, USA
i get the overall feeling that you heard some numbers and are now trying to make the most sense of it.
I had to start somewhere, and used the Samsung EVO m2 as a benchmark guide to compare other SSD's to when I shopped for the ones in my workstation. It seemed to me the information from the vendors was confusing. I can't tell how long these SSD's will last since I have never tracked the I/O flop read-writes -vs- hours/months/years for any drive I have ever owned. 1 million ? 3 million ? Is that 6 months, 4 years ? i do not know - never tracked it.

The best of both Worlds is RAM, something around 5ns (nanoseconds) latency and at best 350GB/s but very NOT affordable in appropriate sizes for storage purposes.
RAM? Are you talking about an in-RAM memory disk, I would make using like 512gb of memory? How does that work - does it read at boot-up - stay in memory and flush on shutdown? Or are you talking about a different SSD technology apart from the M2 or nvme , or sata ?
 

rootgremlin

Member
Jun 9, 2016
42
14
8
I can't tell how long these SSD's will last since I have never tracked the I/O flop read-writes -vs- hours/months/years for any drive I have ever owned. 1 million ? 3 million ? Is that 6 months, 4 years ? i do not know - never tracked it.
It really does not matter because the MTBF (Mean time between failure) is a statistical factor and also something calculated off of failure rates of a given samplerate in a controled environment by the manufacturer.

If a disk has a MTBF of 1 Million hours it is best translated to: "if you have 1 Million drives, every hour one disk will have an unrecoverable error"
see also: Hard disk drive reliability and MTBF / AFR


RAM? Are you talking about an in-RAM memory disk, I would make using like 512gb of memory? How does that work - does it read at boot-up - stay in memory and flush on shutdown? Or are you talking about a different SSD technology apart from the M2 or nvme , or sata ?
If for your usecase the fastest SSD on the market is still too slow (and also if money is of no concern) you could build/buy all kinds of "Exotic" Hardware with nearly unlimited amounts of RAM (also getting bigger every day)
If we are really talking about such "Monster" you would know how to use it.

However, such Systems do not get shutdown on the end of a businessday. The point beeing, they are sized big enough to hold the most vital/hot data in RAM to speed things up.

But basically, yes, a in ram memory disk. But think about it. When you need nanoseconds accesstime an TBs of ram-Storage, how can you access it. If you go Network, this adds massive amounts of latency, so why bother.

see: https://www.quora.com/What-is-the-most-RAM-a-computer-has-ever-had
 
Last edited:

Evan

Well-Known Member
Jan 6, 2016
3,346
598
113
Going off topic but the question of data in memory is a valid one... SAP HANA reloads... just think how much storage performance you need when you have 16TB on ram to fill ;)