NAS build - 32 disk custom cabinet. Freenas.

Discussion in 'DIY Server and Workstation Builds' started by Henrik Lykke Jespersen, Dec 16, 2018.

  1. Henrik Lykke Jespersen

    Joined:
    Dec 16, 2018
    Messages:
    5
    Likes Received:
    7
    Build’s Name: nas01
    Operating System/ Storage Platform: FreeNAS 11.2
    CPU: Intel(R) Pentium(R) CPU G4620 @ 3.70GHz (4 cores)
    Motherboard: SUPERMICRO X11SSL-F
    Chassis: Custom. 5mm polycarbonate. Made to fit specific dimensions.
    Drives: 16 WD Red 3TB, 4 Samsung EVO 860 500GB SSDs, 4 Seagate Baracuda 1TB
    RAM: 16 GB Kingston DDR4 ECC
    Add-in Cards: LSI 3008 SAS3, Intel RAID Expander RES3CV360 36 Port SAS/SATA 12Gb, Intel x550-T2 10Gb dualport ethernet NIC.
    Power Supply: Corsair RMx750 watt PSU
    Other Bits: 10x Corsair ML120 fans, 3.5 inch slot fan controller for 4 sets of fans

    Usage Profile:

    1. Media Storage
    2. System Backups (VMs from Hyper-V, transfer to cloud storage from the NAS for offsite storage)
    3. 1 VM in Bhyve (Server 2019 Domain Controller out of 3 in the AD forest)
    4. Homedrives
    5. PC Image backups
    6. FileHistory backups
    7. iSCSI storage for certain other workloads
    8. Encrypted offsite remote backup endpoint for family and friends
    9. Member of local on-premise AD.

    Other information:

    Background

    Originally I had a QNAP TS-639 which served my needs quite well for many years. However, I grew out of it, and the cost of procuring a bigger one was a quite big onetime investment.

    I was also somewhat hesitant to just expand by replacing disks with bigger ones in the QNAP, as I would eventually be limited by the 6 slots available and I was fearing that the NAS would soon break down (it's around 9 years old now, so...). My concern was also flexibility; 6 or even 12 slots does not give you much room to change the setup if needed (Raid, transforms, expansion)

    Also, by using established brands, you're tied into their ecosystem - i.e. if a PSU, PCB or backplane would break down, I would most likely be forced to purchase an equivalent part at a premium from that vendor. That goes for the software too in many cases (hence the choice of FreeNAS).

    So, first I investigated a 24-slot DAS (Supermicro, xcase.co.uk) to connect to my existing servers. Again, too expensive and somewhat tie-in with vendor.

    I then build my own in plywood with room for 24 disks, which worked quite well - until I had more than 8 disks in it - it simply became too hot (mostly due to some design-flaws in terms of ventilation).

    So, I designed a new one:
    Annotation 2018-12-16 232405.jpg
    Frontview

    Annotation 2018-12-16 232540.jpg
    Backview

    This time with room for 32 disks in two compartments and PLENTY of fans.


    The concept
    Instead of a DAS, I now wanted a dedicated system just for storage. Also, I wanted as good ventilation as possible without too much noise. So, many big and slow fans, where the airflow should be compartmentalized and directed efficiently.

    Three compartments:
    1. Main disk compartment - 24 diskslots
      Push/pull configuration. (3x3 fans)
    2. Server compartment - for the processing and the SAS expander
      Push configuration. (2x fans)
    3. Auxiliary compartment with 8 diskslots, fancontroller and PSU(s)
      Push configuration. (2x fans)
    So far, no disk has gone above 28c under heavy load – at least 10c below the previous plywood cabinet under similar conditions. I have 1mm polycarbonate formed shrouds ready, which will further improve the airflow, by directing the air down in between the disks – if needed, but I don’t believe so, we’ll see in the summer.

    The cabinet is built from clear 5mm polycarbonate. The major parts have been ordered cut to size. The other parts have been cut by myself. Advantage of the polycarbonate is the strength, resistance to cracks and the simple way of working with it. Everything has been glued together with glue made specifically for polycarbonate. Extremely strong bindings!!

    Each disk has its own little bracket on the bottom of the case, preventing the disks from sliding around. In that bracket, 2mm isolating rubber has been glued to bottom to reduce vibrations. There is nearly no vibration sound from the cabinet or the disks.


    The cabinet is designed to fit between a door and the shelf for my servers - 15,5 cm clearance and the NAS is 13,2cm high. The last few centimeters are to be used for a drawer-system underneath the shelf where my servers and backbone switch is located.

    20181216_175013989_iOS.heic[2].jpg

    Also, ventilation exhausts are in the back, designed to direct hot air out and up behind the shelf through some holes in the backside of the shelf (still to be cut).

    Finally
    So the NAS is done – just waiting for some more hardware and the drawer system to be delivered, so that I can pull the NAS out from beneath the shelf and service it from the top.

    At that point I’m installing the 10Gb NIC and direct-connect the ports to my servers (one port for each). Each of the servers also have a direct 10Gb connection with each other (Running Server 2019 Hyperconverged Hyper-V cluster with S2D). All in all, 10Gbit between all the relevant hosts in my network. Connection out from servers and NAS to the rest of the network is bonded 2x 1Gbit.

    For now it’s installed above our washing machine and dryer:

    20181216_210443146_iOS.jpg
    20181216_210447943_iOS.jpg

    Still waiting for the last disks, SSDs and the SAS expander to arrive.

    At that point, the setup will be:

    1. 2x ZFS RAIDZ2 with 8x3TB disks in each (Main storage, 24-disk compartment)
    Or maybe just 2x ZFS RAIDZ1 with 6x3TB and then order more disks for another 6x3TB. Not decided yet.
    2. 1x ZFS Striped+Mirrored with 4x500GB SSDs (Extra VM iSCSI store, 8-disk compartment)
    3. 1x ZFS Striped+Mirrored with 4x1TB disks (Backup of VMs, 8-disk compartment)

    The cabinet has been fully tested in terms of cable management. There is adequate room and it’s quite easy to replace a disk when required. However, to further improve airflow and accessibility, I’m in the process of creating my own power-cable harnesses for the disks, which essentially removes the MOLEX power connectors from the 24-disk compartment, freeing up quite some space (24x molex connecters take up A LOT of real estate).

    So in conclusion.

    1. Very efficient airflow, low noise
    2. Fast NAS!
    3. Low price (materials: about 3.000DKK/450USD – About half to 1/3 the price of a DAS/NAS cabinet)
    4. Plenty of room to expand
    5. Cheap/easy to repair (all standard HW and easy to replace/procure). No vendor lock-in.
    6. Very fun to do! :)

    Very satisfied with the results. Details such as the gluing of the pieces could be a lot better, though. I’m planning to spray-paint the entire cabinet and maybe include a FreeNAD logo near the on/off button in that process. Just for kicks :)
    Hope you enjoyed the reading and maybe got some inspiration!
     
    #1
    Last edited: Dec 16, 2018
  2. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,832
    Likes Received:
    1,489
    cool build!!

    Hopefully you used anti-static poly sheets :)
     
    #2
  3. am4593

    am4593 Active Member

    Joined:
    Feb 20, 2017
    Messages:
    151
    Likes Received:
    31
    solid post, great details. I am curious about two things though. 1 being vibration and 2 being electromagnetic sensitivity. I see this all is setup around a dryer. Are you concerned about the potential vibration or heat put off by the dryer when it runs. Does a dryer create an electromagnetic field? I dont know the answer to that. I assume not, but i dont know. I try to keep all my spinners away from my speakers, which are just giant magnets.
     
    #3
    Last edited: Dec 16, 2018
    T_Minus likes this.
  4. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,832
    Likes Received:
    1,489
    Great point, I didn't even consider that. My dryer is vented direct outside through the wall, no piping\exhaust routing, and there's still light lint \ cotton dust on everything in that room.
     
    #4
  5. PigLover

    PigLover Moderator

    Joined:
    Jan 26, 2011
    Messages:
    2,775
    Likes Received:
    1,116
    Very nice. Like everything about this build - except the part about installing in the laundry. Agree with @am4593 and @T_Minus - lots off issues with the location.
     
    #5
  6. Henrik Lykke Jespersen

    Joined:
    Dec 16, 2018
    Messages:
    5
    Likes Received:
    7
    Thanks for the nice comments and input - much appreciated! :)

    I've had similar concerns in the beginning; @T_Minus and @am4593.
    Let's take it from the top:
    1) Vibration.
    Not an issue at all - new washingmachine - hardly rocks anything. The dryer is old, but makes not vibration either. Also, both of them are on suspension-feet.
    2) Electromagnetic field.
    I've had it measured, using an EMF meter during various cycles of the washingmachine programs. Nothing to measure at all. Granted, I might not have seen the worst-case situations, but still. Low risk tbh.
    3) SED from the PC material.
    Yes, they're anti-statically treated to reduce the risk of such situations. However, from my experience (been involved in showcase computer builds with PC before), non-treated PC isn't THAT much of an issue anymore. Mostly because the electronics are much better today at handling it (sealing, coverage, grounding, etc.). Also, none of the hardware is in direct contact with the PC and have at minimum of 5mm spacing or is standing on an insulating layer of rubber. Not too worried, tbh.

    Also, remember, this is for a limited period only, until I get the drawersystem for it's final mounting below the shelf.
     
    #6
  7. cesmith9999

    cesmith9999 Well-Known Member

    Joined:
    Mar 26, 2013
    Messages:
    1,100
    Likes Received:
    333
    I would still look for a different location than the laundry room.

    Chris
     
    #7
  8. Henrik Lykke Jespersen

    Joined:
    Dec 16, 2018
    Messages:
    5
    Likes Received:
    7
    @cesmith9999 Well, not other option in the house. The garage gets too cold in the winter. I'm NOT setting up all this gear in my office... :)
    Been having all my hardware in the laundry room for the past 2,5 years without any issues at all - being it dust, heat or anything else. It's dry, not too hot, and doesn't disturb the rest of the house. It's also the main drop for all ethernet cables and where all technical installations are located. So, not going to move it... :)
     
    #8
  9. marcoi

    marcoi Well-Known Member

    Joined:
    Apr 6, 2013
    Messages:
    1,334
    Likes Received:
    205
    #9
  10. am4593

    am4593 Active Member

    Joined:
    Feb 20, 2017
    Messages:
    151
    Likes Received:
    31
    funny. We have those screws in our Mac Pro systems, never thought to look for them for use in PCs.
     
    #10
  11. Tooms

    Tooms New Member

    Joined:
    Nov 17, 2018
    Messages:
    29
    Likes Received:
    16
    looks like it is not very easy to change dead drives, do you have to power down and take it apart for that ? or can you easy hotplug the drives ?
     
    #11
  12. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,832
    Likes Received:
    1,489
    They were popular on my antec cases of the last 15 years iirc, I still have a baggy of them to use :D
     
    #12
  13. cesmith9999

    cesmith9999 Well-Known Member

    Joined:
    Mar 26, 2013
    Messages:
    1,100
    Likes Received:
    333
    I do want to say, I would like my own version of this for my office...

    Chris
     
    #13
  14. Henrik Lykke Jespersen

    Joined:
    Dec 16, 2018
    Messages:
    5
    Likes Received:
    7
    @Tooms It's easy peasy. The disks are simply connected with a 8643 to 4x 8482 cable, making it very easy to simply lift the disk, and pull the 8482-connector. And that's hotplug, so no need to shut down the NAS. :)
     
    #14
    Last edited: Dec 20, 2018
  15. Henrik Lykke Jespersen

    Joined:
    Dec 16, 2018
    Messages:
    5
    Likes Received:
    7
    @all reg vibrations.
    The disks rests on a strip of 2mm rubber, which is enough to aborb the vibrations produced. Also, polycarbonate is also fairly good at absorbing vibrations, compared to aluminium and steel for example.
     
    #15
    Last edited: Dec 20, 2018
Similar Threads: build disk
Forum Title Date
DIY Server and Workstation Builds Freenas build, with a rack chassis Yesterday at 1:14 AM
DIY Server and Workstation Builds Dual Xeon video workstation build - worth it? Saturday at 10:09 AM
DIY Server and Workstation Builds New build for a VM server Friday at 11:08 AM
DIY Server and Workstation Builds Client NAS Build Dec 3, 2019
DIY Server and Workstation Builds reconfigure or new build?: Blue Iris and Plex combo box Nov 13, 2019

Share This Page