Client NAS Build

sboesch

Active Member
Aug 3, 2012
390
29
28
Columbus, OH
A client of mine exceeded the capacity of his Synology 5bay and 8bay shelves. He came to me asking for a solution. His current Synology setup is at 80% capacity, this crazy guy bought 24x 16TB Iron Wolf drives and asked me to build a FreeNas solution around them.

His prerequisite was that I keep it under $2500.00 (Minus Disks) Needless to say, challenge accepted.

Case


NORCO RPC-4224 4U
Or
Chenbro RM23524M2

Motherboard


ASRock Rack X470D4U

Processor

AMD Ryzen 7 2700X Processor with Wraith Prism LED Cooler

Power Supply

CORSAIR HX1000
or
Yet undetermined server power supply

Memory

4x Crucial 16GB DDR4-2666 ECC UDIMM

Power Splitters

6x Manhattan, 6" SATA 15-Pin Male to Dual 4-Pin Molex Female Y Splitter, 308977


Host Bus Adapters (Disk Controllers)

LSI Broadcom SAS 9300-8i
LSI Broadcom SAS 9305-16I

SAS Cables (Disk Cables)

6x CableCreation Internal Mini SAS HD Cable, 1.6FT Mini SAS SFF-8643 to Mini SAS 36Pin SFF-8087 Cable, Mini SAS 36Pin to SFF-8643 Cable, 0.5M

M.2 Boot Drive

ADATA SU800 256GB M.2 2280 SATA 3D NAND Internal SSD


Networking

Chelsio 110-1106-30-A0 S320E-CR Dual Port 10gbe PCI-e Standard Profile Adapter
 

BeTeP

Well-Known Member
Mar 23, 2019
490
306
63
Last edited:

Rand__

Well-Known Member
Mar 6, 2014
4,436
864
113
I'd second/third (or where ever we are) a SM Chassis. You probably could get away with a SAS3 EL backplane alleviating the need to get lots of HBAs
(4800 MB/s for 4 SAS lanes can drive a lot of SATA drives
-Mirror => 24 drives in read=200MB/Read per disk max
-Raid Z2 (eg 3 x 8 (6+2) drives gives max 18 data drives =267MB/s per disk)
but I see those might be out of budget new;)

But honestly, it looks like your moving too fast.
Have you identified what his requirements are? (Speed, concurrent users, backup capabilities/requirements [significantly more data potentially?],) ...
Is replacing the Syno the only goal or have there been issues with that box that the new one should address? Any plugins /virtualization on the old box that should run on the new one?
 

sboesch

Active Member
Aug 3, 2012
390
29
28
Columbus, OH
I'd second/third (or where ever we are) a SM Chassis. You probably could get away with a SAS3 EL backplane alleviating the need to get lots of HBAs
(4800 MB/s for 4 SAS lanes can drive a lot of SATA drives
-Mirror => 24 drives in read=200MB/Read per disk max
-Raid Z2 (eg 3 x 8 (6+2) drives gives max 18 data drives =267MB/s per disk)
but I see those might be out of budget new;)

But honestly, it looks like your moving too fast.
Have you identified what his requirements are? (Speed, concurrent users, backup capabilities/requirements [significantly more data potentially?],) ...
Is replacing the Syno the only goal or have there been issues with that box that the new one should address? Any plugins /virtualization on the old box that should run on the new one?
This is home storage device, the dude is a complete data hoarder, I've never seen a TV and movie collection on NAS like this guy has. 10GB is plenty fast for connectivity, he'll rarely saturate it. As for datasets, I'll probably do a two 10 vdev raidz1x2 pools and some hot spares. I have found that raidz1x2 outperforms raidz2 by a good margin, and you use 72% versus 69% of your storage. I am going to have another chat with him about changing chassis, so we will see.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,269
548
113
This is home storage device, the dude is a complete data hoarder, I've never seen a TV and movie collection on NAS like this guy has. 10GB is plenty fast for connectivity, he'll rarely saturate it. As for datasets, I'll probably do a two 10 vdev raidz1x2 pools and some hot spares. I have found that raidz1x2 outperforms raidz2 by a good margin, and you use 72% versus 69% of your storage. I am going to have another chat with him about changing chassis, so we will see.
If this is a storage server mainly for media why even go ZFS? Why not use a non-striped array setup such as Unraid or SnapRAID+MergefFS?
 

Rand__

Well-Known Member
Mar 6, 2014
4,436
864
113
I am going to have another chat with him about changing chassis, so we will see.
It sounds as if even a SAS2 EL backplane would be fine - for the price of a new box you can get a nice older SM chassis, replace all moving parts and get spare's to boot.
 
  • Like
Reactions: sboesch

IamSpartacus

Well-Known Member
Mar 14, 2016
2,269
548
113
384TB of RAW storage and only 64GB of RAM? I haven't used FreeNAS in a few years but is the 1GB of RAM per 1TB of usable storage space no longer their recommendation?
 
  • Like
Reactions: maze

sboesch

Active Member
Aug 3, 2012
390
29
28
Columbus, OH
384TB of RAW storage and only 64GB of RAM? I haven't used FreeNAS in a few years but is the 1GB of RAM per 1TB of space no longer their recommendation?
1GB to 1TB parity would be pretty overkill, the arc is only caching used or frequently used data, I doubt he's going to rewatch ARC value of movies and tv shows, this dude has 24 spindles, that's oodles of IO for hoarding his shows.
 

Rand__

Well-Known Member
Mar 6, 2014
4,436
864
113
384TB of RAW storage and only 64GB of RAM? I haven't used FreeNAS in a few years but is the 1GB of RAM per 1TB of usable storage space no longer their recommendation?
It never was a *hard* requirement - it always depends on the use case (nr of files in arc, nr of users etc).
O/c more Ram means more cache means faster access, but this does not seem to be a prerequisite here...
 

vangoose

Active Member
May 21, 2019
213
52
28
Canada
I say no to raidz1 for 16TB disk, especially 10 disk vdev.

Personally I would do 11 disk raidz3 vdev * 2 + 2 global hot spare.
 
  • Like
Reactions: T_Minus

BeTeP

Well-Known Member
Mar 23, 2019
490
306
63
You probably could get away with a SAS3 EL backplane alleviating the need to get lots of HBAs
BPN-SAS3-846EL1 will set you back ~$400. But I would rather get the BPN-SAS-846A (nominally it's SAS1, but it's passthrough - so it works for SAS3 as well) for $81 shipped instead. Or you can always get one from @BLinux at 3x the price.
 

Rand__

Well-Known Member
Mar 6, 2014
4,436
864
113
well thats 400 + 1 HBA vs 81 + 3 HBA's (or two if he gets a 16 or one for a -24) ;)
Depends on free slots on the board too - the one he picked only has 2 x8 capable slots...
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
6,974
1,561
113
CA
I say no to raidz1 for 16TB disk, especially 10 disk vdev.

Personally I would do 11 disk raidz3 vdev * 2 + 2 global hot spare.
This.

Don't do RZ1 with that many disks, or 16TB disk, don't do RZ1 at all for the most :D my 02