WD RED drives and small file server w/ ZFS

[zogthegreat]

Hi everyone! Happy New Year! Let's hope that this one is better than the last! Although, let's be honest, the bar is rather low here!

So I'm planning to up grade my home file server. Here's my current hardware:

Asus Z9PR-D12 motherboard with 2 Xeon E5-2620 2.0ghz
ASUS PIKE 2008 RIAD flashed to IT mode
70gb DDR3 ECC
2 Intel S3700 200gb // OS drives in mirror RAID - just ordered the drives this month
1 Intel S3700 400gb // SLOG drive - just ordered it this month
1 Nvidia GTX 960 TI // video pass through for Windows VM
1 Radeon RX 570 4GB // Video passthrough for OSX VM
5 port USB 3.0 PCI-e card for VM passthrough
4 port USB 3.0 PCI-e card for VM passthrough

So I'll start with that I understand that my gear is "old" in computer terms, however it does what I need and it's paid for. These days money is tight for everyone, soooooo.....

The next stage of my upgrade will be to change my storage drives. Currently I have 6 Seagate consumer 2TB drives in a ZFS RAIDZ, with an additional 2 2TB drives as cold spares. I want to upgrade to enterprise hard drives and I noticed that the 4TB Western Digital Red hard drives are very reasonably around $60 USD each, so I started doing some research and then I found out about the whole "Oh? Did we forget to mention that these were shingled drives?" problem. Here's a couple of links detailing the problem:

https://www.servethehome.com/wd-red-dm-smr-update-3-vendors-bail-and-wd-knew-of-zfs-issues/

WD Red SMR Drive Compatibility with ZFS - iXsystems, Inc. - Enterprise Storage & Servers

Thanks to the FreeNAS community, we have uncovered a potential ZFS compatibility issue with some capacities of newer WD Red drives that use Shingled Magnetic Recording (SMR) technology. Western Digital’s WD Red hard drives are commonly used in FreeNAS Minis due to good affordability and low...

www.ixsystems.com

So my question is, for a small setup like I'm planning, with the max of 2 users at any time for the SAMBA file server, are these drives "OK" or should I save up some more money and get some WD Gold drives. I'm seeing these going for around $80 - $100 USD, so significantly more for 8 drives.

Also, are there any firmware update for the WD Red's that help with the issue, or would I be better off just getting more consumer hard drives and using something other than ZFS?

Thanks!

zog

 

[zogthegreat]

I think that I've found a simple solution to my question. There are 8TB Seagate EXOS 7e8 for sale on eBay for $120 USD with shipping. Same price per TB as the WD Red's and I will use less drive for the same amount of storage space.

 

[JBond]

you can also try to find some 4TB EFRX, a few months ago I was still able to find them on amazon but am not sure.

 

[andrewbedia]

Just check to make sure any red drives you buy have "EFRX" at the end of the serial number and you're fine. If you're buying new, newegg labels them as SMR/CMR appropriately

regular Seagate drives (Barracuda) seems to be flaky at best from my experience. If you have their NAS drives (VN in the serial number), those have been proven to be reliable.

On the used market, you can usually find HGST/Seagate/WD/Toshiba SAS disks for less than the SATA counterparts and they will outperform red drives by a pretty large margin. The cons to buying these types of drives are noise (seeking is relatively loud), heat, and power consumption; however, I've been pretty happy with dumping NAS drives in favor of NL SAS disks for my personal server.

 

[i386]

So my question is, for a small setup like I'm planning, with the max of 2 users at any time for the SAMBA file server, are these drives "OK" or should I save up some more money and get some WD Gold drives.

With all the "scandals" about the nas and 24/7 hdds (smr, 5400 rpm class) I started to use sata datacenter hdds where I need a lot of storage...
Enterprise hdds > consumer/prosumer oriented hdds; wd gold > wd red

Same price per TB as the WD Red's and I will use less drive for the same amount of storage space.

Is this the only important metric?
What about TB per watt?
Noise? (I think the noise of 24 hdds seeking is more annoying than the spinning fans of the server/psus)

 

[Evan]

I would spend a little and find some say 14tb disks and mirror or even 4 x 8tb hgst he drives for cheap and raidz2 if you want best security and you already have more slave than today. Usually for capacity at a home scale simple use the minimum drives possible and save the power,noise,heat etc. if you need performance use SSD.

(Yes there is some places number is spindles counts in enterprise space with spinning disks but that pretty unique these days, confined to cheap S3 style object store, backup use instead of tape or maybe NAS replication targets)

 

[Samir]

Hmmm....

8 x 4TB = 32TB, 8 x $60 = $480 WD Red
8 x 4TB = 32TB, 8 x $80 = $640 WD Gold

2 x 16TB = 32TB, 2 x $300 = $600 Seagate Exos (same as WD Gold)

If having only 2x drives is an issue, then 8TB drives are an option.

 

[zogthegreat]

Hey everyone, thanks for the advice.

Right now I'm leaning towards HGST 8TB SAS drives, (HUH728080AL4205). They are selling on eBay for $109.00 USD with free shipping.

HGST 8TB on eBay

Since my budget is tight these days, I can start buying one drive a month for four months and create a 16TB, (actually 13.62TB), RAID-Z2 array. My Asus PIKE2008 supports SAS and I ordered some SATA to SAS adapters from China.

In the meantime, I will transfer my current 7TB RAID-Z2 to my Supermicro X8SIL motherboard that has a Dell PERC 700 in IT mode. Right now I have two cold spares for the array and 2TB drives can be found for $15 - $20 if any of the spares go bad.

Hmmm....

8 x 4TB = 32TB, 8 x $60 = $480 WD Red
8 x 4TB = 32TB, 8 x $80 = $640 WD Gold

2 x 16TB = 32TB, 2 x $300 = $600 Seagate Exos (same as WD Gold)

If having only 2x drives is an issue, then 8TB drives are an option.

@Samir , where do you find the WD drives at that price? Thus far, the WD Red and Gold drives seem to be going at around $125 + USD.

I would spend a little and find some say 14tb disks and mirror or even 4 x 8tb hgst he drives for cheap and raidz2 if you want best security and you already have more slave than today. Usually for capacity at a home scale simple use the minimum drives possible and save the power,noise,heat etc. if you need performance use SSD.

(Yes there is some places number is spindles counts in enterprise space with spinning disks but that pretty unique these days, confined to cheap S3 style object store, backup use instead of tape or maybe NAS replication targets)

@Evan

(chuckle) As far as the "power, noise, heat", I'm living in an apartment with "free" electricity. My X8SIL "server" is in my storage locker on my balcony where it is currently -5c/23f. I'm running a program called "Security Monitor Pro" with 4 cameras watching some pigeons and their chicks who made a home on my balcony for the winter. Currently, my CPU is running about 60% on all cores and the CPU temperature is 60c with stock fan. I'm actually thinking of switching to watercooling to bring the CPU temps down.

It's called Covid boredom, I'm sure others here are in the same spot right now!

I looked into your suggestion, but 14TB drives are out of my reach right now, so I'm taking your advice and going with 4 8TB SAS drives.

 

[zogthegreat]

So I thought that I would follow up with what I went with. I got 2 x's 200GB Intel DC S3700's for my OS and a 400GB Intel DC S3700 for my SLOG drive. I also got 2 x's Seagate 8TB SAS drives, (ST8000NM0065), for data storage. Although right now it's more storage than I need, I'm planning ZFS for data storage, so I can expand the array later.

Side note, whilst deciding how much storage that my budget could handle, I recalled the Seagate ST-412 10mb drive that I purchased back in 1983 for my IBM PC XT. At the time, everyone asked me, "What are you going to do with all of that storage space?"! The times, they are a changin!

I crammed everything into an old Corsair mid-tower case that I laying around and I'm now in businesses!

Thanks to everyone for their help and suggestions.

zog

 

[Samir]

Nice setup!

@Samir , where do you find the WD drives at that price? Thus far, the WD Red and Gold drives seem to be going at around $125 + USD.

I didn't get a notification in time to remember what search led me to those prices--at this point I have no clue. I think it was one of the sites that specializes in server drives--or maybe I used your numbers--can't recall at all.

 

[Evan]

Sounds like a really useful setup.
There was a time when everybody was hoarding data but with streaming so easy now I don’t hoard tv and movies any more. Do have all my music stored as lossless FLAC but even with my massive collection that’s less than 1TB.

 

[zogthegreat]

I'm planning to run several VM's on this setup, Win 10 w/ video and USB passthroughs, OS X, also with video and USB passthroughs, a NextCloud server, because lossless FLAC may only take up 1TB, but my 15 year old daughter with her iPhone camera can hit the multi TB stage real fast!

Plus, I gave Apple $1000 for her bleeping phone, I don't want to keep paying for storage!

The nice thing about the Asus board is it has 1 16X slot and 6 8X slots, so I can have multiple GPU's and USB cards for my passthroughs. I'm using an Asus PIKE 2008 RAID card, so I don't lose any slots for an HBA to handle the SAS drives. I was looking at getting a PowerEdge R720, but keeping the noise levels at a reasonable level can be a PITA. This setup gives me more flexibility as far as cooling, I'm running 6 Antec 120's, (3 in, 3 out), and everything running around 40c, with the CPU's running at 55c at idle with Stock aluminum coolers. I'll either upgrade the stock coolers to some Cooler Master Hyper 212's or go to water cooling.

 

[Whaaat]

The nice thing about the Asus board is it has 1 16X slot and 6 8X slots, so I can have multiple GPU's

Don't try to install more than 2 full height GPU in this motherboard to avoid burning out of 24-pin connector. Multiple GPUs-capable mobo will have at least one additional molex onboard.