ZFS storage server upgrade (pool capacity)

lpallard

Member
Aug 17, 2013
216
6
18
So I am contemplating an upgrade of my Freenas ZFS pool capacity by upgrading the hard drives to larger ones. Classic case but I am trying to do it as cheaply as possible without getting into reliability issues associated with refurbished/used drives.

Some limitations:
The chassis supports 16 drives but due to electricity usage and re-silvering times, I'd like to maintain the number of drives as low as possible. I am currently using 8 drives.

I need a usable pool size of 22TB. This is based on a detailed usage calculation and I assume I would have enough space for at least the next 5 years or more...

Out of the 8 drives I use, I currently have three 3TB drives and five 2TB drives in the pool. To minimize costs I thought of reusing the 3x 3TB drives. A good idea? I am confused because here in Canada, prices are all over the place and are in average 25-35% higher than in the US..... I guess thats the price for living in the Great North (TM) :)

If I keep the existing 3TB drives, a possible scenario according to the STH RAID Calculator, would be to get another 9 drives for a total of 12 in RAID-Z3 which will give me 24.6TB pool usable space.

So some questions:
The following configurations will provide a pool size of about 22TB:
6 drives of 8TB (21.8TB) = Astronomical cost (around $2800), lets forget that ;)
8 drives of 6TB (27.3TB) = A bit excessive capacity and insanely expensive ($2400)
10 drives of 4TB (25.5TB) = Again insanely expensive and a bit larger than necessary ($1960)
12 drives of 3TB (24.6TB) = A bit larger than necessary but cheapest option so far (already own 3 drives) ($1280)

Which is best?

Why is storage so expensive !!!!??? Damn 4TB drives came out in 2011 and are still overpriced.
 
Last edited:

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
6,992
1,568
113
CA
I'm taking a guess that your pricing is "new" in Canada because those prices are about 2x used/ebay prices for drives here in the US, and Canada should only cost you ~100$ more total to have 10 drives shipped to you.

checkout this seller:

5TB $120 Shipped - New (5000GB) 5TB 5700RPM 64MB Cache SATA 6.0Gb/s 3.5" Hard Drive -FREE SHIPPING

6TB - $150 shipped - New (6000GB) 6TB 5700RPM 64MB Cache SATA 6.0Gb/s 3.5" Hard Drive -FREE SHIPPING

They have 1 YR Warranty, and I believe those are WD RED -- if not they have others that are "white label red" or others that are more $$ and more power that are RE white label, etc...

Those 6TB are 1/2 your estimated cost, provide a lot of room to grow, and likely affordable shipping or even free if you PM them and tell them you want to buy 8 of them. You could go raidz2 if you thought that was safe enough and jump capacity too.
 
  • Like
Reactions: lpallard

Patrick

Administrator
Staff member
Dec 21, 2010
11,906
4,868
113
We should be at 12TB drives in the not so distant future which will start to push prices down.

You are not going to like this idea but what if you upgraded using 2x 6TB drives in RAID 1 or 4x 6TB drives in a striped set of mirrors?

The advantage is that you spend less than half. By adding sets of RAID 1 drives you get an easy way to continue expanding.
  • 4x 6TB drives = 12TB usable - if you need more
  • 3x 3TB drives = 3TB usable with a hot spare available
15TB usable in 6 data drives (and one spare)

Targeting 5 years is a bit forward looking. I generally look at a 3 year estimate at most. In 3 years you can upgrade to larger drives at lower price points. The other side is that by the time you hit 5 years there is a good chance the 2TB/ 3TB existing drives you have are going to see a higher failure rate due to age. I think I have stopped using all 2TB drives at this point for that reason.
 
  • Like
Reactions: lpallard

lpallard

Member
Aug 17, 2013
216
6
18
I'm taking a guess that your pricing is "new" in Canada because those prices are about 2x used/ebay prices for drives here in the US, and Canada should only cost you ~100$ more total to have 10 drives shipped to you.
Yes, all prices for WD Reds on newegg.ca

My guess based on ordering a s**t ton of stuff from the US is that for a single hard drives, shipping will be anywhere between 30 & 60$...... Nevertheless I take that you guys trust these "white labels" drives on ebay??

You could go raidz2 if you thought that was safe enough and jump capacity too.
I thought about moving to a RAID Z2 setup instead of Z3. I initially opted for Z3 because I only could backup a fraction of the pool and thought my data is >>>>> than speed so Z3 it was. Now, Z2 would free up some TB's and change the equation. What are you guys doing?

Lets say Number of drives, setup (RAID Z, Z2, Z3, 1+0, etc) and importance from 1 to 10 (10 being you'd rather have your house burn down ;) )

I like Patrick's idea of several vdevs (or volumes in Freenas language?). With several clusters of lesser drives, upgrading will be less costly (cheaper to upgrade 2 or 4 drives in a vdev than all the pool at once) and yet will give flexibility.

Perhaps I could run the existing 6 2TB drives in 6 RAID1 vdevs for usable of 6TB, then another vdev of 2x 3TB for 3TB usable, then another vdev of 2x 6TB for another usable 6TB. This would provide 15TB usable and allow me to upgrade one group of 2 drives at the time when $$$ allows.

Can I do that with Freenas?

Side question:

To help me move forward, for a 4 drive setup that would be only used for periodic backups (4x per year) and would NOT be online 24/7, would a RAIDZ (RAID 5) be OK? I know on Freenas's forums, they are going crazy with RAID 5 because its terrible blablabla.. I totally get the idea of reliability but is it still holding up for very light/casual use?
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
6,992
1,568
113
CA
For my Home AIO I run RaidZ2 for my primary storage with 6x WD RED 5TB, and then my backup storage (3x5TB) is 3x mirrored.
 

kroem

Active Member
Aug 16, 2014
245
40
28
35
Just want to chim in and mention that you will see performance degrade from when you've filled the pool 60ish% and upwards. (may not even matter if it's just media storage)

The setup with multiple different sizes striped to one big pool will "work", but bear in mind that if you lose two "wrong" drives you'll lose the entire pool.

I'm foolish enough to run periodic backups to a raidz :)
 

lpallard

Member
Aug 17, 2013
216
6
18
Would a HGST white label drive for HP works with M1015 controllers?

6TB HGST OEM By HP UltraStar He6 SATA 7200RPM Hard Drive HUS726060ALA640

What about a no brand drive like this:

New 6TB 7200RPM 128MB Cache SATA III 6Gb/s 3.5" Security CCTV DVR Hard Drive

What about WD blue drives? THey are cheaper by a significant margin over the others...

The setup with multiple different sizes striped to one big pool will "work", but bear in mind that if you lose two "wrong" drives you'll lose the entire pool.
Not sure why I hadnt considered this.... So instead of lets say 6 vdevs of 2 disks in RAID1, I would use 3 vdevs of 4 disks in RAIDZ2. That way i would have to lose 3 drives in a vdev to lose the entire pool. Better than RAID1 IMO.... Plus upgrading a single vdev to larger drives would be easier.
 

rubylaser

Active Member
Jan 4, 2013
842
229
43
Michigan, USA
Would a HGST white label drive for HP works with M1015 controllers?

6TB HGST OEM By HP UltraStar He6 SATA 7200RPM Hard Drive HUS726060ALA640

What about a no brand drive like this:

New 6TB 7200RPM 128MB Cache SATA III 6Gb/s 3.5" Security CCTV DVR Hard Drive

What about WD blue drives? THey are cheaper by a significant margin over the others...


Not sure why I hadnt considered this.... So instead of lets say 6 vdevs of 2 disks in RAID1, I would use 3 vdevs of 4 disks in RAIDZ2. That way i would have to lose 3 drives in a vdev to lose the entire pool. Better than RAID1 IMO.... Plus upgrading a single vdev to larger drives would be easier.
Just be aware that those particular HP HE6 drives have no warranty. I bought one from that seller before they changed the listing to include that they are HP OEM drives vs. HGST drives that come with five year warranty. They did accept a Best Offer of $180, but I returned it for it not matching the item description.

I'd buy these instead that are not OEM drives, so they should have the balance of the 5 year warranty for the same price.

6TB GST Ultrastar He6 6.0Gb/s, 7200 RPM,3.5" (0F18335) HDD HUS726060ALA640
 
  • Like
Reactions: Patrick

lpallard

Member
Aug 17, 2013
216
6
18
The WD Reds you pointed out on Amazon are over $350CAD by the time they arrive at my doorstep... Insane.

There is a guy selling WD blues (6TB) New and is about 30 minutes from my place in Canada. I wonder for backing up only (and NOT for 24/7 use in my freenas server) if they would work well? Blues are the slow ones from WD? Nobody (or very little) people on freenas forums are using Blues so that gets me worried but again, I am still using Greens on my main pool so... ;)