NAS build - 32 disk custom cabinet. Freenas.

[Henrik Lykke Jespersen]

Build’s Name: nas01
Operating System/ Storage Platform: FreeNAS 11.2
CPU: Intel(R) Pentium(R) CPU G4620 @ 3.70GHz (4 cores)
Motherboard: SUPERMICRO X11SSL-F
Chassis: Custom. 5mm polycarbonate. Made to fit specific dimensions.
Drives: 16 WD Red 3TB, 4 Samsung EVO 860 500GB SSDs, 4 Seagate Baracuda 1TB
RAM: 16 GB Kingston DDR4 ECC
Add-in Cards: LSI 3008 SAS3, Intel RAID Expander RES3CV360 36 Port SAS/SATA 12Gb, Intel x550-T2 10Gb dualport ethernet NIC.
Power Supply: Corsair RMx750 watt PSU
Other Bits: 10x Corsair ML120 fans, 3.5 inch slot fan controller for 4 sets of fans

Usage Profile:

1. Media Storage
2. System Backups (VMs from Hyper-V, transfer to cloud storage from the NAS for offsite storage)
3. 1 VM in Bhyve (Server 2019 Domain Controller out of 3 in the AD forest)
4. Homedrives
5. PC Image backups
6. FileHistory backups
7. iSCSI storage for certain other workloads
8. Encrypted offsite remote backup endpoint for family and friends
9. Member of local on-premise AD.

Other information:

Background

Originally I had a QNAP TS-639 which served my needs quite well for many years. However, I grew out of it, and the cost of procuring a bigger one was a quite big onetime investment.

I was also somewhat hesitant to just expand by replacing disks with bigger ones in the QNAP, as I would eventually be limited by the 6 slots available and I was fearing that the NAS would soon break down (it's around 9 years old now, so...). My concern was also flexibility; 6 or even 12 slots does not give you much room to change the setup if needed (Raid, transforms, expansion)

Also, by using established brands, you're tied into their ecosystem - i.e. if a PSU, PCB or backplane would break down, I would most likely be forced to purchase an equivalent part at a premium from that vendor. That goes for the software too in many cases (hence the choice of FreeNAS).

So, first I investigated a 24-slot DAS (Supermicro, xcase.co.uk) to connect to my existing servers. Again, too expensive and somewhat tie-in with vendor.

I then build my own in plywood with room for 24 disks, which worked quite well - until I had more than 8 disks in it - it simply became too hot (mostly due to some design-flaws in terms of ventilation).

So, I designed a new one:

Frontview


Backview

This time with room for 32 disks in two compartments and PLENTY of fans.


The concept
Instead of a DAS, I now wanted a dedicated system just for storage. Also, I wanted as good ventilation as possible without too much noise. So, many big and slow fans, where the airflow should be compartmentalized and directed efficiently.

Three compartments:

1. Main disk compartment - 24 diskslots
   Push/pull configuration. (3x3 fans)
2. Server compartment - for the processing and the SAS expander
   Push configuration. (2x fans)
3. Auxiliary compartment with 8 diskslots, fancontroller and PSU(s)
   Push configuration. (2x fans)

So far, no disk has gone above 28c under heavy load – at least 10c below the previous plywood cabinet under similar conditions. I have 1mm polycarbonate formed shrouds ready, which will further improve the airflow, by directing the air down in between the disks – if needed, but I don’t believe so, we’ll see in the summer.

The cabinet is built from clear 5mm polycarbonate. The major parts have been ordered cut to size. The other parts have been cut by myself. Advantage of the polycarbonate is the strength, resistance to cracks and the simple way of working with it. Everything has been glued together with glue made specifically for polycarbonate. Extremely strong bindings!!

Each disk has its own little bracket on the bottom of the case, preventing the disks from sliding around. In that bracket, 2mm isolating rubber has been glued to bottom to reduce vibrations. There is nearly no vibration sound from the cabinet or the disks.


The cabinet is designed to fit between a door and the shelf for my servers - 15,5 cm clearance and the NAS is 13,2cm high. The last few centimeters are to be used for a drawer-system underneath the shelf where my servers and backbone switch is located.



Also, ventilation exhausts are in the back, designed to direct hot air out and up behind the shelf through some holes in the backside of the shelf (still to be cut).

Finally
So the NAS is done – just waiting for some more hardware and the drawer system to be delivered, so that I can pull the NAS out from beneath the shelf and service it from the top.

At that point I’m installing the 10Gb NIC and direct-connect the ports to my servers (one port for each). Each of the servers also have a direct 10Gb connection with each other (Running Server 2019 Hyperconverged Hyper-V cluster with S2D). All in all, 10Gbit between all the relevant hosts in my network. Connection out from servers and NAS to the rest of the network is bonded 2x 1Gbit.

For now it’s installed above our washing machine and dryer:




Still waiting for the last disks, SSDs and the SAS expander to arrive.

At that point, the setup will be:

1. 2x ZFS RAIDZ2 with 8x3TB disks in each (Main storage, 24-disk compartment)
Or maybe just 2x ZFS RAIDZ1 with 6x3TB and then order more disks for another 6x3TB. Not decided yet.
2. 1x ZFS Striped+Mirrored with 4x500GB SSDs (Extra VM iSCSI store, 8-disk compartment)
3. 1x ZFS Striped+Mirrored with 4x1TB disks (Backup of VMs, 8-disk compartment)

The cabinet has been fully tested in terms of cable management. There is adequate room and it’s quite easy to replace a disk when required. However, to further improve airflow and accessibility, I’m in the process of creating my own power-cable harnesses for the disks, which essentially removes the MOLEX power connectors from the 24-disk compartment, freeing up quite some space (24x molex connecters take up A LOT of real estate).

So in conclusion.

1. Very efficient airflow, low noise
2. Fast NAS!
3. Low price (materials: about 3.000DKK/450USD – About half to 1/3 the price of a DAS/NAS cabinet)
4. Plenty of room to expand
5. Cheap/easy to repair (all standard HW and easy to replace/procure). No vendor lock-in.
6. Very fun to do!

Very satisfied with the results. Details such as the gluing of the pieces could be a lot better, though. I’m planning to spray-paint the entire cabinet and maybe include a FreeNAD logo near the on/off button in that process. Just for kicks
Hope you enjoyed the reading and maybe got some inspiration!

 

[T_Minus]

cool build!!

Hopefully you used anti-static poly sheets

 

[am4593]

solid post, great details. I am curious about two things though. 1 being vibration and 2 being electromagnetic sensitivity. I see this all is setup around a dryer. Are you concerned about the potential vibration or heat put off by the dryer when it runs. Does a dryer create an electromagnetic field? I dont know the answer to that. I assume not, but i dont know. I try to keep all my spinners away from my speakers, which are just giant magnets.

 

[T_Minus]

solid post, great details. I am curious about two things though. 1 being vibration and 2 being electromagnetic sensitivity. I see this all is setup around a dryer. Are you concerned about the potential vibration or heat putoff but that when it runs. Does a dryer create an electromagnetic field? I dont know the answer to that. I assume not, but i dont know. I try to keep all my spinners away from my speakers, which are just giant magnets.

Great point, I didn't even consider that. My dryer is vented direct outside through the wall, no piping\exhaust routing, and there's still light lint \ cotton dust on everything in that room.

 

[PigLover]

Very nice. Like everything about this build - except the part about installing in the laundry. Agree with @am4593 and @T_Minus - lots off issues with the location.

 

[Henrik Lykke Jespersen]

Thanks for the nice comments and input - much appreciated!

I've had similar concerns in the beginning; @T_Minus and @am4593.
Let's take it from the top:
1) Vibration.
Not an issue at all - new washingmachine - hardly rocks anything. The dryer is old, but makes not vibration either. Also, both of them are on suspension-feet.
2) Electromagnetic field.
I've had it measured, using an EMF meter during various cycles of the washingmachine programs. Nothing to measure at all. Granted, I might not have seen the worst-case situations, but still. Low risk tbh.
3) SED from the PC material.
Yes, they're anti-statically treated to reduce the risk of such situations. However, from my experience (been involved in showcase computer builds with PC before), non-treated PC isn't THAT much of an issue anymore. Mostly because the electronics are much better today at handling it (sealing, coverage, grounding, etc.). Also, none of the hardware is in direct contact with the PC and have at minimum of 5mm spacing or is standing on an insulating layer of rubber. Not too worried, tbh.

Also, remember, this is for a limited period only, until I get the drawersystem for it's final mounting below the shelf.

 

[cesmith9999]

Also, remember, this is for a limited period only, until I get the drawersystem for it's final mounting below the shelf.

I would still look for a different location than the laundry room.

Chris

 

[Henrik Lykke Jespersen]

@cesmith9999 Well, not other option in the house. The garage gets too cold in the winter. I'm NOT setting up all this gear in my office...
Been having all my hardware in the laundry room for the past 2,5 years without any issues at all - being it dust, heat or anything else. It's dry, not too hot, and doesn't disturb the rest of the house. It's also the main drop for all ethernet cables and where all technical installations are located. So, not going to move it...

 

[marcoi]

maybe just install some anti-vibration orings on the hdds?
Ie something like this: https://www.amazon.com/Akust-Hard-Drive-Vibration-Screw/dp/B010D13AFC

 

[am4593]

funny. We have those screws in our Mac Pro systems, never thought to look for them for use in PCs.

 

[Tooms]

looks like it is not very easy to change dead drives, do you have to power down and take it apart for that ? or can you easy hotplug the drives ?

 

[T_Minus]

funny. We have those screws in our Mac Pro systems, never thought to look for them for use in PCs.

They were popular on my antec cases of the last 15 years iirc, I still have a baggy of them to use

 

[cesmith9999]

I do want to say, I would like my own version of this for my office...

Chris

 

[Henrik Lykke Jespersen]

@Tooms It's easy peasy. The disks are simply connected with a 8643 to 4x 8482 cable, making it very easy to simply lift the disk, and pull the 8482-connector. And that's hotplug, so no need to shut down the NAS.

 

[Henrik Lykke Jespersen]

@all reg vibrations.
The disks rests on a strip of 2mm rubber, which is enough to aborb the vibrations produced. Also, polycarbonate is also fairly good at absorbing vibrations, compared to aluminium and steel for example.