SFF NAS Upgrade - ZFS Build Advice

[nitrobass24]

Thought I would elicit some advice from the group regarding an upgrade I am working on.

Current Setup:
Synology 1812+

• 4x8TB RAID5 + 845 DC Evo 960GB as read cache for Media Storage (The spinning drives will be repurposed for an offsite backup)
• 2x PM853t 960GB Raid1 for NFS ESXi and Docker data store
• 2x 1GBe LACP

Future Setup will be based on a DS380 w/ Asrock C2550 ITX board ASRock Rack > C2550D4I
I have also purchased 32GB of ECC RAM for it.

I am planning on buying new drives for the media storage and moving to ZFS on FreeNAS 9.10

What i am confused on is the best path forward in terms of zPools and vDevs.
Should I buy 4x10TB Raidz1 or 6x8TB Raidz2? Both will cost me about the same. The 8TB path provides more space and redundancy, but at the expense of future expandibility/flexibility.

Lastly I am a bit torn on how to put my SSDs to use. In terms of SSDs at my disposal:

• 1x 845 DC Evo 960gb
• 2x PM853t 960gb
• 1x Toshiba M.2 256 GB (no PLP)

I could use my SSDs as an L2ARC and put my VM datastore on the same zPool with the media or I could forgo a large L2Arch and use them to setup a sepearate vdev and zpool?
Or i could sell them all and buy different SSDs altogether.

 

[Rand__]

L2ARC is "slow" cache to increase read speed, depending on how much of your data is actually non-ram cached it might help or not.
you will want a slog device (eg s3710) for a zfs/nfs based VM storage.
RaidZ1 is not recommended any more as it only has a single redundant disk and if a second fails during a long resilvering action on a 10TB disk (I think my 8TB drives resilvered 16hrs) your pool is toast -> at least z2
Alternatively you can go Mirroring - better for vm's (higher IOPs) but less space. Technically a 2 disc failure can kill the pool as well if it hits the wrong disks though.

You can use the 853Ts as vm share if 500 MB is enough space (960GB, mirrored, zfs utilization <60% for best performance)

 

[chief_j]

I went the iscsi route with esxi because even with slog, my vm performance over nfs was not the best.
I could never get nfs performance above 50MB/s over nfs with vmware using 4x1tb in a mirrored pool.
iscsi I get around 500MB/s

 

[nitrobass24]

I went the iscsi route with esxi because even with slog, my vm performance over nfs was not the best.
I could never get nfs performance above 50MB/s over nfs with vmware using 4x1tb in a mirrored pool.
iscsi I get around 500MB/s

Makes sense since iSCSI is natively asynchronous. Will have to test both for sure!

 

[Patrick]

@nitrobass24 you may have 1-2 more SSDs than you think. I was just informed by my helper that some may have been left before they packed and sent it off.

I am a big fan of ZFS mirrors. It also makes adding capacity simple.

 

[PigLover]

6x8GB RaidZ2 for the main pool (agree with the comment on avoiding Z1 when using large capacity drives). Don't bother with Zil or L2ARC - its a media and backup pool after all.

Use the little m.2 as you boot drive and configure a small pool on it for misc. stuff.

Use the three Samsung 96oGB drives as a "fast pool" for VMs and other work (well, these days only sorta fast...). Either do a 2 way mirror of the 845ts or use all three in a Z1 pool to get 1.8GB out of it. Back it up to the big pool regularly. Again - no need for Zil or L2ARC.

Of course whatever additional SSDs Patrick found for you might change this.

 

[nitrobass24]

Thanks @PigLover

I am going to go with 6x 8TB drives, should also help with keeping re-silvers down to a reasonable amount of time.
I was going to boot from a SanDisk Ultrafit 16GB I have laying around. Was thinking of using the M.2 NVME as L2ARC for the VM Datastore, but I could see how that might be sub-optimal and reduce my overall ARC performance. Might instead move it into my Desktop at home.

I only actively use about 250gb for VMs anyways. Just got the 960GB for the extra NAND. Maybe the move is to sell all of the SSDs and get some more NVMe. I'll wait to see what I have when I get the box from Patrick.

 

[nitrobass24]

Been doing a lot of reading on the internet and I am now considering going to Mirrored vDevs.

Single RAIDz2 6x 8TB = 32GB RAW
3x Mirrors of 2x 8TB = 30GB RAW

Most of this data is archive / cold storage of Media, so write performance is not critical. If my understanding is correct resilvering is done at the vDev level and not the zPool level. Resilvering 10TB will be about 1/3 the time if done this way. Seems like a no brainer to sacrifice 2TB of RAW capacity.

Am I missing something?

 

[PigLover]

3x mirrors of 2x 8TB drives yields 24TB raw, not 30. Each mirror pair yields an 8TB vdev, and striping 3x8TB vdevs is 24TB.

Perhaps you meant 3 mirrors of 10TB drvives?

There is also small difference in resiliency between a Raidz2 of 6 drives and 3x 2-way mirrors. In the Raidz2 you can lose any two drives and still preserve your data. With the 3x 2-way mirrors you can lose three drives (better) but only if they are exactly the right three (one drive from each mirror). If you lose both drives in any one mirror then the pool is kaput.

On resilver the mirror probably does finish faster - it basically becomes a simple replication of one drive. But if you have an unrecoverable error on that one drive you are reading from then again - kaput. With the Raidz2 if you lose a 2nd drive the pool is still in-tact enough to complete a resilver.

 

[nitrobass24]

Ahh yes, you are right...Math

Makes sense on the additional redundancy.


Sent from my iPhone using Tapatalk

 

[_alex]

if not too late:
consider expanding over the redundancy, too
(google for ,the hidden cost of zfs' or similar)

with the raidz2 you can only expand by adding another vdev (of preferably the same size/number of disks).
with the mirrors its same, but this means you only need 2 hdd to expand your pool.

 

[nitrobass24]

Ended up with 6x 8TB raidZ2

 

[fractal]

if not too late:
consider expanding over the redundancy, too
(google for ,the hidden cost of zfs' or similar)

with the raidz2 you can only expand by adding another vdev (of preferably the same size/number of disks).
with the mirrors its same, but this means you only need 2 hdd to expand your pool.

I realize he already went with 6 drives in raidz2 but .. He would have had the same capacity with a smaller incremental upgrade cost if he went with the same drives in two raidz1 vdevs. It would survive half of the two drive failure permutations.. Expansion would require three drives for another three drive vdev.

Not sure if it is a good or bad idea but it was something I pondered when thinking about the same question.

 

[_alex]

yes, 2x raidz1 had been an option, too.
besides issues when upgrading (now a vdev of 48tb ) also time for resilvering will be much longer than with mirrors.
i wouldn't go raidz at all if somehow simple mirrors work, for this reasons.
But when capacity matters that much maybe the z2 was the right choice.