Correct way to set up storage for home.

[TLN]

Hello.

Looking forward to build by AIO server to replace HP Microserver. Now it's time to set up a proper storage.
I'm planning on running ESXi (most likely). One VM will work as NAS for me. What is correct way to connect all the drives?
It's obvious that ESXi goes onto flash drive, separate SSD for VM and so on.

1. Do i need external RAID controller?
2. What's current best practive: bigger drives or more drives? I'm looking for ~6-10TB storage. I can do it with 5x2TB drives, or 2x6TB drives. What's best practice here?
3. What's most common RAID level? I'm looking into RAID-5/6(if 5 drives or more).
4. How do I connect drives to VM? Passthrough the controller to VM. Create RAID and allocate space for .vmdk?



Thanks.

 

[ttabbal]

The most common practice for an all-in-one is to use an HBA which is passed through using vt-d to the NAS VM. The most popular cards for this are reflashed IBM M1015, and Dell H210/H310. They get flashed to "IT Mode" which disables the on-card RAID functions, allowing the NAS to use ZFS to manage them. This is usually FreeNAS, Solaris, or Linux. With Solaris based setups, napp-it is a very popular option.

How many and which drive sizes to use is a matter for debate. There are pros and cons to both strategies. Many smaller drives is often cheaper, can perform better depending on many factors. Downside is you need lots of bays to hold them. Larger drives cost more, and can be less reliable as they are newer technology and thus there is less data on them.

Which RAID level is part of the above... They are all tradeoffs. RAID6 (raidz2 in ZFS) allows any 2 drives to die with no data loss. At the total storage level you are looking to do, I wouldn't do RAID5 (ZFS raidz1). The time it takes to rebuild the array can cause a second failure. Note that failure modes are better with software based setups on ZFS. You are more likely to lose a block due to bad sectors than having the whole disk die at once. ZFS can recover from this, many hardware RAIDs cannot as they don't know which disk has the bad block. Performance with both of these parity RAID setups is about equal to the performance of the slowest disk in the array. You can combine arrays in a stripe to improve performance. The ZFS raidz setups can do pretty well in streaming sequential I/O, so for media storage they aren't bad.

For home use, I prefer stripped mirrors, what you might see called RAID10. Each disk is backed up with a mirror, the whole lot are striped into one large array. There are a couple benefits. Performance for random I/O is significantly improved. Disk replacements are much faster as you don't need to read n-1 disks to rebuild the mirror, and there is less overall data to read as you aren't reading the whole array to calculate the missing data. Expansion is also much easier. Add or replace 2 drives, free space goes up. The downside is obvious, storage efficiency is 50%.

At the end of the day, it's your data, machine, power bill, etc., so you have to decide how to want to do it. There's not really one "correct" way.

 

[T_Minus]

Hello.

Looking forward to build by AIO server to replace HP Microserver. Now it's time to set up a proper storage.
I'm planning on running ESXi (most likely). One VM will work as NAS for me. What is correct way to connect all the drives?
It's obvious that ESXi goes onto flash drive, separate SSD for VM and so on.

1. Do i need external RAID controller?
2. What's current best practive: bigger drives or more drives? I'm looking for ~6-10TB storage. I can do it with 5x2TB drives, or 2x6TB drives. What's best practice here?
3. What's most common RAID level? I'm looking into RAID-5/6(if 5 drives or more).
4. How do I connect drives to VM? Passthrough the controller to VM. Create RAID and allocate space for .vmdk?



Thanks.

1. For AIO most don't use a RAID controller. Some may still because they have it, but they don't pass it through to ZFS they would use it for guest os datastores in esxi.

2. I think this depends on if you have any drives, future expansion plan/thoughts, drive capacity in chassis, cost factor, performance, data importance, etc... With 2 drives you can only do a mirror with 3 you can do a mirror but on a 3rd disk as well. Yet, if you did 10x 2TB you'd have 10TB if you do a pool of mirrors which would yield much > performance than a 2 or 3 disk mirrored set.

3. Depends on your specific needs. For 3 disks I would (and I do) do a mirror setup. For 6 disks I'm doing RAID6 (Z2), and for 24 disks I'm doing a pool of mirrors.

4. In ESXI you pass-through the controller (LSI HBA most common) to the "NAS" (storage VM), and from within that VM you provide access back to ESXI. With Napp-It/OmniOS it's common to setup a NFS share, and then locate that within ESXI configuration and configure it as a datastore.

Keep in mind you most likely cannot pass-through the "onboard" SATA controller, but you can pass through the onboard LSI HBA if your motherboard has one.

 

[TLN]

@ttabbal

I totally understand there's no correct way, and every case is a bit different.
Mainly I had following questions:
1. Internal controller or external - EXTERNAL.
2. Hardware RAID or Software - TBD
3. RAID levels - 10 is better then 5/6.

I still have doubts about FreeBSD with ZFS or proven OMV, which works great for me. When I was testing it previously - OMV got better results then FreeBSD(Both NAS4Free, FreeNAS).

Initially I was thinking about 4-5 drives in RAID, which is good idea for big storage.
Now I'm thinking about two RAID1. I already have couple of 3TB drives, I might get another pair. Any reason to run RAID 10 instead RAID 1 + RAID 1?

 

[T_Minus]

RAID10 is not "better" than raid5 or 6 they both have Pros and Cons.

What does "better results" mean to you? Easier to use? Faster to setup? Better performance?

rAID1 + raid1 (separate arrays/pools) doesn't have the performance from striping as RAID10 (pool of mirrors) will.

 

[TLN]

1. For AIO most don't use a RAID controller. Some may still because they have it, but they don't pass it through to ZFS they would use it for guest os datastores in esxi.

2. I think this depends on if you have any drives, future expansion plan/thoughts, drive capacity in chassis, cost factor, performance, data importance, etc... With 2 drives you can only do a mirror with 3 you can do a mirror but on a 3rd disk as well. Yet, if you did 10x 2TB you'd have 10TB if you do a pool of mirrors which would yield much > performance than a 2 or 3 disk mirrored set.

3. Depends on your specific needs. For 3 disks I would (and I do) do a mirror setup. For 6 disks I'm doing RAID6 (Z2), and for 24 disks I'm doing a pool of mirrors.

4. In ESXI you pass-through the controller (LSI HBA most common) to the "NAS" (storage VM), and from within that VM you provide access back to ESXI. With Napp-It/OmniOS it's common to setup a NFS share, and then locate that within ESXI configuration and configure it as a datastore.

Keep in mind you most likely cannot pass-through the "onboard" SATA controller, but you can pass through the onboard LSI HBA if your motherboard has one.

1. Pass HBA to VM responsible for storage. Other VM get access via Network.
Guest OS stored on SSD drive.

2. I see difference in performance. We're talking about 4-6 drives total. I don't see a reason to have either 10drives+ at home, neither 2drives only. I
Obviously it will be like: Small SSD for VM, Bigger SSD for VM, Drives for storage via HBA.

3. As expected. Reasonable amount to go Raid6(raidz2) is 6 drives at least.

4. Have to google about Napp-it/OmniOS.

But I can pass onboard HBA? If there's any drawbacks compared to PCI controller?

 

[ttabbal]

How many drives just depends on how much space you want and how much performance you want. Adding more mirrors or raidz's to the pool == more performance. My goal was to saturate 10Gbe, not sure I've made it, but it gets pretty close. Not that the goal was a really reasonable thing to want, it is one of those "because I can" things. I also wanted more space. Then a good deal came up for 2TB drives, so I have a bunch of them in the server now. More important for me, I have lots of free space. I was getting pretty full on the old setup. The server box holds 24 drives, so why not?

It's extremely unlikely you can pass the onboard SATA controller through to a VM without conflicts. And if you did, what would you put VM storage on? You can try to pass individual drives through, but that introduces a whole new layer of complexity and failure points. The cards cost about $50, hardly worth the bother to try to avoid them.

I've used some moderately high end RAID cards. My ZFS box stomps them performance wise, no contest. Add in the error recovery/detection parts, and there is no reason a home user should use hardware RAID in my opinion. There are likely cards that can beat ZFS. But not if you include price/performance as home users tend to. And make sure to buy a spare, in case you have a controller failure. I really like that I can use my array on any hardware that's capable of talking to the drives. I don't have to worry that the latest Adaptec card doesn't support the on-disk layout my older card did.

But like I said, it's your data, you have to decide how you want to store it. I'm just telling you how I store mine and why.

 

[TLN]

But like I said, it's your data, you have to decide how you want to store it. I'm just telling you how I store mine and why.

Pretty much what I want to hear.
It's more obvious what you should do with 10+ drives, but less obvious with smaller amount of drives.
I'd like to have less then 6 drives, and 2 SSDs.

 

[whitey]

I'd like to have less then 6 drives, and 2 SSDs.

Then go 6 drive raidz2 spinners for data/media/gen purpose (CAPACITY stg pool) and mirror of ent class fast/large 960GB or more I would suggest for VM storage (PERFORMANCE stg pool)

Single SC826 or similar chassis or SC216 if you like 2.5" and 24 bays v.s. 12 bays. You start w/ 8 drives and have a lil room to grow w/ the SC826, a LOT of room to grow into the SC216...if you catch 'the sickness' as I call it :-D

As for NAS options the good ones have been spoken to here in this thread. Check my sig for what I entrust my precious data to. :-D although an Illumos based NAS system is a VERY good choice as well for a DIY setup, AIO all day long.

EDIT: Whoops I thought you said 'less than 8 drives' not 'less than 6 drives'... I HIGHLY suggest at least minimum 8 drives in config suggested above to get you off to the races as I cannot recommend 4 drive raidz w/ capacities as they are these days and meeting your requirements unless you can deal w/ 4x 4tb drives and raidz2, but in that case I'd almost recommend going stripped mirror config just for added redundancy/resiliency/performance in my mind at least for same space roughly.

 

[ttabbal]

For ease of use and upgradability, I would do 2x 4tb mirrors. 8TB usable, so it's in your desired range. Decent performance, easy to upgrade later, just add 2 larger drives to the mirror, wait for it to finish reslivering, remove the old ones. That also gets you under the <6 data drives you requested.

I started out with 6 disks in a raidz2. It works well, but speed suffers. The lower space penalty is nice, at first, but to upgrade you need to buy 6 disks, and replace them one at a time, reslivering between each. That gets old fast. Just one of the reasons I use mirrors now.

 

[TLN]

For ease of use and upgradability, I would do 2x 4tb mirrors. 8TB usable, so it's in your desired range. Decent performance, easy to upgrade later, just add 2 larger drives to the mirror, wait for it to finish reslivering, remove the old ones. That also gets you under the <6 data drives you requested.

I started out with 6 disks in a raidz2. It works well, but speed suffers. The lower space penalty is nice, at first, but to upgrade you need to buy 6 disks, and replace them one at a time, reslivering between each. That gets old fast. Just one of the reasons I use mirrors now.

Well, that exactly my thoughts.
I can go with RAID 6 with 6x2Tb drives, that will cost me ~600$. 8Tb of useful space.
Or I can buy 2x4TB drives and set up in RAID 1, this will cost me ~300$. I already have pair 3Tb drives.

Mixing old and new drives (i.e. buying 4x 3Tb drives), which gives me Raid 6 with 6 drives makes any sense?

 

[ttabbal]

Well, that exactly my thoughts.
I can go with RAID 6 with 6x2Tb drives, that will cost me ~600$. 8Tb of useful space.
Or I can buy 2x4TB drives and set up in RAID 1, this will cost me ~300$. I already have pair 3Tb drives.

Mixing old and new drives (i.e. buying 4x 3Tb drives), which gives me Raid 6 with 6 drives makes any sense?

OK, so you have 2x 3TB drives now? Do they contain data you want to keep? Do you want to use them in the new setup?

If you want to re-use them, that's fine. Get 2x 4TB and mirror them, copy the data from the 2x 3TB over to the new mirror. Scrub the array to make sure the new copy is good, then erase the old drives and add them as a mirror to the existing array. Now you have 7TB usable with performance about 2x the single RAID6. Add another mirror for more space and about 3x the RAID6 performance and so on.

Note that I only work with ZFS right now, so while I know that will work on ZFS, I haven't a clue how it might work if you go with hardware RAID. It depends on your card.

You also might consider trying some used drives. Properly burned in and tested, which you should be doing to all drives even brand new from the manufacturer, they work quite well and are likely past the point where you would need to worry about early failure. The drives I got were $32/ea for 2TB. Even new drives should be <$150 for a 4TB. That price seems high to me. Perhaps it's location though.

 

[TLN]

OK, so you have 2x 3TB drives now? Do they contain data you want to keep? Do you want to use them in the new setup? .

I have 2x3TB drives, and I think some 1Tb drives. May be something else.

I'm basically torn between two options:
=Put all drives in big RAID. RAID 6 makes sense with 5-6 drives at least. So I'm buying 4 more drives, and running RAID 6 via hardware controller.
=Have several pairs of mirror drives. That's pretty much what I have now. That will be cheaper, but does look a bit worse though.

I can do this with ZFS or RAID, it's not a deal-breaker for me.

I'd jump on bunch of used drives for $32 per 2Tb. As I mentioned, that gives 8Tb from 6 drives in RAID 6, which is plenty. I also expect pretty good performance here, correct?

 

[ttabbal]

I'm not sure what you mean by "look a bit worse". Is it about total vs usable space? I've decided to just not worry about it. The trade is that I get to upgrade cheaper and easier by upgrading 2 drives vs. 6 drives. As a home user, I have limited funds to put into it, so that alone is worth a fair bit to me.

Performance for a RAID6 is about as good as the slowest drive in the array. So about the same as a single drive. If that's "pretty good performance" depends entirely on you. It was good enough when I started, but as I added more clients, it couldn't keep up. Raw throughput was ok most of the time, but many clients hitting different files started to lag even for media playback. This was with 2x 6-disk arrays. With the mirrors, my I/O wait time is significantly improved and overall performance is far better. Most importantly, I don't get stutter and lag while watching movies, even with every player in the house on different shows. Priorities and all that.

For 2TB drives, this is where I got mine.

Hitachi Ultrastar A7K2000 HUA722020ALA331 2TB 32MB cache Internal Hard Drive

They bumped the price up, $40/ea now. Not as good, but not terrible either.

Say you picked up some of those. You could make 2 mirrors for 4TB, copy the data from your old drives, add them in for 7TB total. Then when prices come down, or you just need more space, upgrade 2 of the 2TBs to 4TB drives, giving you 9TB total, and so on.

If you did 6 of them in RAID6, you get 8TB usable now. But if you want to upgrade for more space, you have to replace all 6 before you see any added space available. This is why I think mirrors are more cost efficient in the long run for home use.

I should note that there are some setups that are more flexible. Unraid is one I hear about that people like. There are 2 reasons I don't use it. First is cost, they charge for it. Which is their right, but when enterprise level software is free, I don't see the point. It also has less of a track record. ZFS has years of enterprise deployment behind it, with little trouble reported. ZFS is also open source and very compatible. My old array started on Solaris, moved to Linux, back to Solaris, then to BSD (freeNAS), now Linux again. No problems doing that. With Unraid, I'm stuck with their platform. When I started, I wouldn't have guessed that I would change the OS around that much, but I did, and it was nice to be able to. They all have their pros/cons.

 

[TLN]

I'm not sure what you mean by "look a bit worse". Is it about total vs usable space? I've decided to just not worry about it. The trade is that I get to upgrade cheaper and easier by upgrading 2 drives vs. 6 drives. As a home user, I have limited funds to put into it, so that alone is worth a fair bit to me.

No, it's not about usable space at all. I might get away with 4-5Tb of data. 8TB is plenty for me.
Saying "looks worse" I mean:
For 2 drives RAID1 is only option.
For 10 drives RAID6 is great way to go.
5-6 drives exactly in the middle. It's where you should go from simple home consumer solution to higher end. From soft raid - to dedicated RAID controller.

Performance for a RAID6 is about as good as the slowest drive in the array. So about the same as a single drive. If that's "pretty good performance" depends entirely on you. It was good enough when I started, but as I added more clients, it couldn't keep up. Raw throughput was ok most of the time, but many clients hitting different files started to lag even for media playback. This was with 2x 6-disk arrays. With the mirrors, my I/O wait time is significantly improved and overall performance is far better. Most importantly, I don't get stutter and lag while watching movies, even with every player in the house on different shows. Priorities and all that.

That make some sense, and that's pretty much what I currently have.
I expected to get better performance with RAID6, rather then slowest disk. I mostly interested in READ, I can wait while my photos are copied

Say you picked up some of those. You could make 2 mirrors for 4TB, copy the data from your old drives, add them in for 7TB total. Then when prices come down, or you just need more space, upgrade 2 of the 2TBs to 4TB drives, giving you 9TB total, and so on.

If you did 6 of them in RAID6, you get 8TB usable now. But if you want to upgrade for more space, you have to replace all 6 before you see any added space available. This is why I think mirrors are more cost efficient in the long run for home use.

Have to do math now.
6 drives and 8 TB of space is good, but it there's no performance increase - what's the point? Have to do homework now )

 

[TuxDude]

No, it's not about usable space at all. I might get away with 4-5Tb of data. 8TB is plenty for me.
Saying "looks worse" I mean:
For 2 drives RAID1 is only option.
For 10 drives RAID6 is great way to go.
5-6 drives exactly in the middle. It's where you should go from simple home consumer solution to higher end. From soft raid - to dedicated RAID controller.

Better to look at it this way...

For higher performance (especially write performance), go RAID 1, at the cost of capacity efficiency
For higher capacity / efficiency go parity-raid (RAID 5 up to 6 to 8 disks, RAID 6 after that), at the cost of horrible random-write performance.

Also keep in mind that ZFS does things differently from pretty much every other raid stack on the planet. Where a Raid-Z2 (ZFS-based raid-6) will perform like a single drive, any other raid-6 (md software raid, any hardware raid) will read at the speed of N-2 disks, and do not bad at sequential writes, but have a horrible penalty for random writes (which will keep the disks busy seeking and impact your read performance as well). So for eg. a media server with a ton of read-mostly storage, RAID-6 is great. For holding a bunch of VMs though, RAID-6 will keep the disks busy doing random writes and everything will be slow.

That make some sense, and that's pretty much what I currently have.
I expected to get better performance with RAID6, rather then slowest disk. I mostly interested in READ, I can wait while my photos are copied

Then consider a RAID-6 implementation other than ZFS. You won't get the automatic checksum verification and could be vulnerable to silent data corruption, but you will get good capacity and also good read performance.


Or perhaps consider snapraid - similar to unraid which was mentioned earlier in this thread, but is fully open-source. Pro's and con's to it as well, but its what my media server is running and for that kind of usage its pretty good. Every file is only stored on a single disk, so the performance to read/write a single file is the same as what a single disk would give - only if multiple files on different disks are in use would you possibly use the performance of multiple drives. On the other hand when you want to watch a movie only a single drive has to spin-up, while the others all remain in power-saving mode. It also allows mixing drive capacities, as long as your parity disk(s) are >= the biggest drive(s) in the array. And it supports anywhere from 1 to 6 parity disks - 1 parity would be similar to raid-5 (or raid-z), 2 parity would be like raid-6 (or raid-z2), etc. Plus if you ever have more failures than what the parity can automatically repair, you don't lose all of the data on the entire array - only the data on the failed drives is lost.

 

[Keljian]

Speed wise you're wrong. I have an 8 drive raidz2 which reads and writes at circa 550MB/s (may actually be limited by network- long story).

In fact, it beats the snot out of my former mdraid raid-6 setup for speed. (Which peaked at 300ish MB/s and often was worse)

This is more than sufficient for home use.

 

[ttabbal]

raidz does well at sequential access. Sometimes better than mirror setups. But it falls on its face with random I/O. A media server with one client can be sequential. Add more clients and it becomes random.

In most cases, it is probably good enough performance for home use. Where it fails for home use in my mind is upgrades. If you start running low on space, stripped mirrors can upgrade 2 drives. Your 8 drive raidz2 needs 8 drives upgraded. That might not be an issue for some people. But most home users I know would have a tough time getting 8 drives at once, but 2 isn't too bad.

 

[TLN]

Well, with RAID6 I'll get a performance upgrade anyway, right?
Idea of buying bunch of used 2Tb drives is tempting, but I still have doubts about it.

 

[TuxDude]

Where it fails for home use in my mind is upgrades. If you start running low on space, stripped mirrors can upgrade 2 drives. Your 8 drive raidz2 needs 8 drives upgraded. That might not be an issue for some people. But most home users I know would have a tough time getting 8 drives at once, but 2 isn't too bad.

All of the RAID-Z variants (Z2, etc.) have that issue, but it is a lot less of a problem with other parity raid implementations.

With md raid-6, you can add drives one at a time to the array and restripe across them - an 8-drive array becomes a 9-drive array. If your hardware can support hot-adding the drive(s) then you can grow a md array with no downtime at all. You can also online convert between raid levels, etc. with md - you could start with a 2-drive mirrored array, add 2 more drives and grow to a 4-drive raid-10 array, and then add another 2 more drives and migrate to a 6-drive raid-5 array, add 2 more and move to a 8-drive raid-6.

btrfs has issues with parity raid still, so I wouldn't advise using its raid-5/6 mode for data you care about - but it also supports online grow/shrink by any amount of drives as well as online conversion between raid levels. And on top of that it also supports using drives of mixed capacity - go ahead and make a btrfs-raid-1 array from your existing two 3TB drives, then as you need more space add a single 4TB drive to the array, a few months later add a 6TB to it, next year add a pair of 8's. Btrfs-raid-1 will turn that mix of drives into about 16TB of usable capacity.

And again snapraid is about as flexible as can be - add/remove drives of any capacity, anywhere from 1 to 6 parity disks, etc. But being a batch process instead of real-time raid it has lots of other trade-offs - its great for media servers, not an option at all for VM hosting.