What's the "right" configuration with RAID-Z?

[mstroud]

It feels like I am getting closer to the end of my build process for a new NAS running ZFS, so I am asking for a little bit of help to get it all wrapped up. This question is specifically about which drive + RAIDZ configuration I should aim for with this box.

Originally I thought that I would have two 5 disk ZFS RAID-Z setups that were mirrors of each other. After running some benchmarks and seeing the performance (or not) of just one 5 disk setup I'm not sure sure that doubling up on the disks is such a good idea. The hope is to get better safety and redundancy without sacrificing too much in the way of performance. That way, when a disk inevitably dies, the entire NAS doesn't spiral down the drain and I'm not frantically running to the local stores looking for replacement drives.

Previously, I had a single hardware RAID-5 enclosure running 5 disks formatted XFS directly attached to a Proxmox VE (Debian) server box. This setup was the "primary" storage and was made up of 1TB disks. I then had a second, separate storage PC running FreeNAS 7 and ZFS on top of five 2TB disks. This was my "secondary" storage and it was mounted via NFS to the server box. A weekly rsync kept the two in close parity to each other. I had many, many disk failures on the primary box, possibly due to my choice of early Samsung 1TB disks. To my knowledge, there have been no failures on the secondary box, with WD 2TB green disks.

Performance on the primary box was excellent, as it was directly attached to the server. Transfer speeds on the secondary box were terrible, partly due to the leftover hardware I built that system from, and partly from FreeNAS 7. My new storage server is large enough to pull all this storage into a single enclosure on much more modern hardware, so I hope that these performance issues are gone.

My workload for these boxes include general PC backups, music/video/photo storage, DLNA serving, VM storage and the eventual need for a 24/7 file sync server (for when Windows Live Mesh goes away later in 2012). My network clients are a mix of Windows 7, Linux, PS3, Google TV and Android devices. Any thoughts or configurations you could share would be very helpful in making my decisions.

Thanks in advance.

 

[gea]

a good summary for pool design:

http://www.nex7.com/readme1st
http://constantin.glez.de/blog/2010...ove-oracle-solaris-zfs-filesystem-performance

I would always define needed security level first and build pools accordingly
ex: if i may allow any two disks to fail without data loss, i would
build pools from n x 3 x mirror vdevs (expensive but ultra fast) or
build pools from n x Raid-Z2 vdevs

and add enough RAM, the more the faster (used for caching)

if your vdevs are build from more than say 8 disks, i would use raid-Z3 due to long rebuild time.
For a mainly backup and media server, i would build a pool from one raid-z2 vdev (up to 7 disks), if more from one raid-z3 vdev

For database, webserver or ESXi datastores, i would use always mirrors

ps
it is not possible to mirror vdevs, they are always striped

 

[mstroud]

That's some helpful data, thank you. However, even though I've read through all that information I'm still a bit puzzled on what to do.

One article seems to say multiple Z1 vdevs are the way to go. So I should do two sets of 4x2TB RAID-Z1?
Another article indicates that a single, large Z2 vdev is the way. So that means I should do a single 8x2TB RAID-Z2?

Life sure was simpler when I had less information.

 

[mmmmmdonuts]

Mirroring and striping is typically used for speed applications (traditional Raid 10), such as VM datastores, databases and other uses where there are many concurrent reads and writes are occurring at once.

If I am reading what you really want to do is have more of a media server. Traditionally with a media server you don't care to much about the speed, unless you have 10GBe connections because a 8 disk Raid Z2 is going to saturate a 1GB link anyways. The downfall to this is that if you want to upgrade the amount of space on the disks your going to have to replace 8 disks rather than doing 4 disks at a time. If you stripe the two 4 disk Raid-Z's together you will get double the read speed, but you greatly increase your chances of losing the pool if a disk goes down due to resilvering time and the additional disk stresses that occur on the pool with a failed disk. In general most people tend to recommend staying away from Raid-Z, because if a disk goes down you are very prone to losing another.

Bottom line is if I was making the decision I would go with the 8x2TB Raid-Z2.

 

[mstroud]

Thanks for that practical and clear comparison! So the 8x2TB Z2 is a single zpool, single vdev setup? Do I need to fiddle with ZFS folders or datasets? I'm using NAPP-IT so this is all new to me. Now I'm really showing my lack of ZFS knowledge, sadly.

I was leaning toward the 8x2TB setup but I wasn't sure if the speed would be an issue. Given that the real reasons behind this new NAS purchase & build are all about stability and redundancy, I needed that push to switch to RAID-Z2. I'll have to try to switch over to that soon... after I backup the data I've already loaded onto my 5 disk RAID-Z configuration.

 

[mmmmmdonuts]

So the 8x2TB Z2 is a single zpool, single vdev setup?

Yes.

Do I need to fiddle with ZFS folders or datasets?

First what you do is you create a zpool. From the zpool you create ZFS folders (in a sense a dataset is the same thing as a ZFS Folder) which act like shares. So my setup is something like this:

zpool : data

zfolders: Music, Videos, Pictures, Backup, etc.

They would be mounted /data/Music

I recommend reading both of gea's manuals already on napp-it.org if you haven't already.

 

[mstroud]

Got it. I have it all setup now and will be posting more information once I get things stabilized. Thanks for all the help!