[Feedback sought] Enthusiastic NAS build [4.5y usage update 2019-12]

Discussion in 'DIY Server and Workstation Builds' started by MacLemon, Feb 17, 2015.

  1. MacLemon

    MacLemon New Member

    Joined:
    Feb 16, 2015
    Messages:
    20
    Likes Received:
    19
    See posting #52!
    --
    I'm working on a somewhat enthusiastic home NAS system and would like your feedback on my choice of parts.

    Here's the details so far, reasonsoning why I chose these is below.
    Build’s Name: Work in progress
    Operating System/ Storage Platform: FreeNAS/FreeBSD
    CPU: Intel® Atom™ C2750
    Motherboard: SuperMicro A1SAM-2750F
    Chassis: Fractal Design NODE 804
    Drives: 6 × Seagate ST4000DM000 Data Sheet (PDF), Product Manual (PDF)
    RAM: This RAM doesn't fit as pointed out by BlueLineSwinger. 2 × 16GB Samsung DDR 3 1600 CL 11, ECC Reg. 1.35V
    Add-in Cards: 1 × IBM ServeRAID M1015 (2 × 4 Port SATA/SAS HBA)
    Power Supply: TBD
    Other Bits:
    • A small-ish SATA SSD (probably Samsung EVO).
    • 2 × Mini-SAS SFF8087 to SATA breakout cables. (SAS8087OCF-06M)
    • A bunch of cables that I've missed to list here
    • Cable ties, velcro, to keep things tidy.
    • Standard power cord
    • UPS to cope with power dips/surges, to cleanly shut the box down in case of a power failure.
    Usage Profile:
    Mostly classical “Home NAS” usage, like storage of audio/video, other random files. (AFP, SMB/CIFS)
    Plex- & Firefly (mt-daapd/iTunes) server. Must be capable of live transcoding two full HD video streams simultaneously. (CPU supports SSE/VMX but I don't expect Plex to use it at all.)
    Local backup target for OS X's TimeMachine.
    Other remote machine's backups over ssh/rsync each to their own FreeBSD jail.
    Some Jails/VMs for mixed software/OS services & testing. (CPU is VT-x/EPT capable.)
    ZFS-RAID Z2 for the main storage pool connected to the HBA.
    ZFS-RAID Z1 for further backup storage (probably by recycling a few existing spindles I can free up once this box is in production) connected to the onboard SATA 2 ports.
    Encrypted storage supported by AES-NI.
    Probably the usual things you forgot that only come to mind after you have ordered parts. What could that be?

    Intentions:
    Low energy consumption when idle yet enough power for the power user in me. Silent operation, as few fans as possible, as many as needed. Large fans to keep the rpms down and the noise frequency low. I prefer good quality components over cheap ones but don't want to spend unnecessary amounts.
    IPMI interface goes to my management LAN.
    2 × Gbit/s ethernet ports trunked for file access
    Remaining 2 ethernet ports used for jails/VMs, likely on different VLANs. (TBD)
    HBA goes into the PCIe 8 × slot. The 4 × slot remains empty for now.
    I intend to run the HDDs in acoustic managed mode for silent operation given the performance impact is worth it.
    No gfx or audio on this box.

    Open questions:
    • Which ATX power supply to get? Should be very silent and energy efficient. (Eyeing an 80 Plus Gold or Platinum model) Will 450W be ok-ish, even if I decide to add more HDs later up to a maximum of 10 (LFF) and 2 (SFF) which fit in the chassis? (ATX PSU up to 260mm fits.) I also need to get power to all the spindles.
    • Is it likely that I need more fans than the 3 × 120mm fans that come with the chassis? (It can accomodate up to 10 fans but I don't intend to build a hovercraft.)
    • I'm looking into getting a TPM 1.2 module for storing SSH host keys of the box. Any suggestions what TPMs work with FreeBSD?
    • I'd like to connect my hardware random number generator to one of the USB ports. (preferably an internal port.)
    • This bullet intentionally left blank.
    • Have I missed something totally obvious? I'm not that used to building my own boxes.
    Why I chose these components:
    Feel free to disagree and suggest better components and please tell me why!

    CPU: The Atom C2750 has 8 cores (8 threads, no hyperthreading) and provides plenty of multithreading speed yet has very low power consumption and support for up-to 64GB RAM. (ZFS and VMs looove RAM, though I'll start with 32GB.)

    Why not these CPUs?
    • Atom C2758: Supports QuickAssist which is less suitable to general server tasks but better for network packet handling. C2758 vs. C2750
    • Atom C2550: 4 Cores/4 Threads version. About half the multithreading performance which would still be enough for video transcoding. 5W less power consumption under full load, not much of a difference when idle. C2750 vs. C2550
    • Xeon E3-1230 v3: Significantly higher power consumption. Limited to 32GB RAM. (I intend to upgrade to 64GB RAM within a year.) E3-1230 vs. C2750
    RAM:
    1.35V modules use less power than 1.5V modules and stay cooler leading to better energy efficiency and more silent operation. Can save between 4W (idle) and 8W (loaded) with 4 × 16GB DIMMs unscientifically extrapolated from this comparison of 1.5V/1.35V RAM on this forum. Modules as speced by SuperMicro for the chosen motherboard.

    Motherboard:
    Why not these MBs?
    • ASRock Rack C2758Di: A tempting alternative at a good price point. It sports additional 4 × SATA 3 and 2 × SATA 2 ports providing enough SATA ports. They are powered by Marvell SE9230, SE9172 which are known for poor throughput and stability problems. Only 2 × Gbit Ethernet ports + IPMI. ASRock does not have any reputation in the server market.
    • SuperMicro A1SAi-2750F: Pretty much identical. mITX form factor opposed to µATX. Hence only a single PCIe 8× slot. Only actual downside being the use of SO-DIMMs which are more expensive and hard to get. Since there is enough space in the chassis for a µATX board I prefer the one listed at the top.
    • SuperMicro A1SA7-2750F: This board would pretty much be the perfect fit with its 16 × SATA 3 ports powered by an LSI 9116 controller. Since it uses a proprietary form factor which doesn't fit in any standard chassis it would have required use of a SuperMicro chassis which are great in the data center, but acoustically inappropriate for home use.
    HBA SAS/SATA Controller:

    Why not this one?
    • LSI SAS 9211-8i: This would be my HBA of choice The chosen IBM controller is identical to this one. Its firmware can be crossflashed. The LSI original is about twice the price for the same product. They are based on the LSI 2008 controller chip.
    • A gazillion other LSI PCIe controllers…
    Since the motherboard only has PCIe 2.0 slots there is no point in getting a PCIe 3.0@8× HBA (based on the LSI SAS 2308). Also the most limiting factor for my setup will be Gbit ethernet. I really don't expect to run into any HD performance issues here. If need be I can still add two SSDs as L2ARC/ZIL to accelerate ZFS storage pools.

    The LSI SAS 2008 RAID Controller/ HBA Information thread here has been of tremendous help for me! Thanks to Pieter!

    Disks
    Based on experiences by Backblaze with tons of disks, building their Storage Pod 4 and also my personal experience.

    Why not this one?
    • HGST HDS5C4040ALE630 (Datasheet (PDF)): I'd actually prefer this drive over the Seagate model. I mostly rejected the HGST disks because I already have 3 Seagates I can use and mixing brands is not the smartest idea. The Seagate disks are about 20% cheaper than HGST.
    Any other questions or suggestions?
    MacLemon
     
    #1
    Last edited: May 22, 2015
    Kristian, Patrick and NeverDie like this.
  2. Chuckleb

    Chuckleb Moderator

    Joined:
    Mar 5, 2013
    Messages:
    1,017
    Likes Received:
    330
    For a power supply, I just bought this for my server and don't know how well it will work. $109 from eBay. It's silent, 500W platinum, could be fun. Has a single 8pin for CPU power so can't do dual CPUs, but that won't be an issue for you:

    Rosewill SilentNight Series SilentNight-500 500W,80 PLUS PLATINUM Certified,Fan-less,Modular Design,Single +12V Rail,ATX12V v2.3/EPS12V v2.92,SLI Ready,CrossFire Ready,Active PFC Power Supply - Retail - Newegg.com

    Since you're going 4TBs, you could get the white label 4TB drive for about $135 or so I think, maybe less. They are enterprise drives, faster but run hotter. There are also some white label 4-6TBs that could bring the cost (and heat) down. Either way, the NAS drives work well, run a bit slower, but bring the heat (and power) down. Case fans can suck up 1-2W each (I measured last night on my bench computer) but then can be set to spin down when idle so that's cool.

    I'm looking at rebuilding my file server as well but want an Atom with more PCIe slots. I have a few cards to shove in (RAID, 10GbE, eSATA, etc).
     
    #2
  3. Patriot

    Patriot Moderator

    Joined:
    Apr 18, 2011
    Messages:
    1,288
    Likes Received:
    662
    I would still buy HGST drives and just replace the seagates as they fail out...
    Which in my experience will be rather quickly for just three drives... and is in line with backblaze.
    There is a reason backblaze's average age of seagates is just 1.4 mo. ...
    Edit 1.4yrs not mo.
     
    #3
    Last edited: Feb 18, 2015
    Kristian likes this.
  4. NeverDie

    NeverDie Active Member

    Joined:
    Jan 28, 2015
    Messages:
    299
    Likes Received:
    26
    Extremely nice write-up! Looks as though you may have pushed up against the 1000 character posting limit, but you wrote so well I wouldn't have minded reading even twice as much.

    The A1SAM-2750F that you linked to looks like it has a fan on the CPU, so don't get that one. Since you want quiet, get the big passive one instead.

    As to whether mixing brands of HDD's is unwise or not: is there actual evidence that it's unwise, or is it just a theoretical argument that's been repeated so much it's accepted as fact? I'm really curious to know.
     
    #4
  5. TuxDude

    TuxDude Well-Known Member

    Joined:
    Sep 17, 2011
    Messages:
    615
    Likes Received:
    336
    If you're dealing with high-bitrate high-resolution video (eg. blu-ray images / remux's / 1080p @ 40mbps or more) then I question whether that will be enough CPU for real-time transcoding of even a single stream, let alone two at once. I would look into a haswell-based Xeon for a job like this - still very good idle power but far more performance available when you need it.
     
    #5
    BlueLineSwinger likes this.
  6. NeverDie

    NeverDie Active Member

    Joined:
    Jan 28, 2015
    Messages:
    299
    Likes Received:
    26
    This one seems like it might be ideal for your purposes. I haven't been able to find it for than than $500 though, so it comes at a premium. Your main objection wasn't price though but the apparent lack of an appropriate case. I bet it would fit in the Nanoxia Deep Silence 6, which would also likely give you the acoustic qualities that you desire that the SuperMicro chassis would lack. IIRC, the latest version comes with space for 16 HDD's, so you can separate whatever HDD's you have to improve airflow/cooling rather than pack them altogether (which is what most people typically do, but which would necessitate running either more fans or running fans at higher RPM (which risks being noisier) and/or the acceptance of higher drive failure rates).
     
    #6
    Last edited: Feb 17, 2015
  7. NeverDie

    NeverDie Active Member

    Joined:
    Jan 28, 2015
    Messages:
    299
    Likes Received:
    26
    Also, if you do opt for a smaller case, like the fractal design 804: I'm pretty sure the Nanoxia Deep Silence 1 has been proven in reviews to be much quieter (last time I looked it was still easily the quietest of anything available, with head-and-shoulders better silencing).
     
    #7
  8. NeverDie

    NeverDie Active Member

    Joined:
    Jan 28, 2015
    Messages:
    299
    Likes Received:
    26
    Also, out of the MB's that you're considering, the passive heatsink of the mini-itx's is shorter and runs hotter than on the micro-atx's or the proprietary board. Just FYI in the event that's a factor in your decision. At the margin, I'd pick the proprietary motherboard or the micro-atx motherboard over the mini-itx for that reason, on the risk-mitigation theory that any required case fan cooling will be easier and less noisy.
     
    #8
    Last edited: Feb 17, 2015
  9. BlueLineSwinger

    BlueLineSwinger Active Member

    Joined:
    Mar 11, 2013
    Messages:
    149
    Likes Received:
    54
    Seconded. I wouldn't want to be restricted by an Atom CPU if I required transcoding.

    The RAM listed won't work. It's registered ECC (RDIMM). The Avotons will only accept unbuffered ECC (UDIMM). So, while the Avotons technically support up to 64GB, you're still effectively limited to to 32GB due to the lack of non-registered 16GB UDIMMs. And I'm really not seeing anything in your description that would require more than 32GB. ZFS in a home setting certainly doesn't. The oft-quoted 1GB RAM per TB of storage is aimed more at office/datacenter setups. Even adding a few containers/jails shouldn't push you past 32GB.

    For the mainboard I'd recommend the Supermicro X10XL7. The included LSI 2308 makes it a really good value. Pair it with an appropriate CPU for your requirements (e.g., E3-1230v3) and RAM from the HCL, and you should be good to go.
     
    #9
    TuxDude likes this.
  10. capn_pineapple

    capn_pineapple Active Member

    Joined:
    Aug 28, 2013
    Messages:
    356
    Likes Received:
    80
    This basically cuts out your need for an external HBA, which should offset the additional costs of having a more expensive yet powerful CPU and gives you more RAM breathing room.
     
    #10
  11. Chuckleb

    Chuckleb Moderator

    Joined:
    Mar 5, 2013
    Messages:
    1,017
    Likes Received:
    330
    @MacLemon Ok, you have to get the power supply I linked to at the top. I just slid it in and it works well. The ports have nice dust covers which I haven't seen before. It's totally silent, nice and heavy. If 500W will do you, this is a great price for a platinum rated power supply. I got a steal for open box on ebay @$109 but it's really nice...
     
    #11
  12. NeverDie

    NeverDie Active Member

    Joined:
    Jan 28, 2015
    Messages:
    299
    Likes Received:
    26
    #12
  13. Hank C

    Hank C Active Member

    Joined:
    Jun 16, 2014
    Messages:
    642
    Likes Received:
    66
    it doesn't have enough PCIE lanes for the e3 chip..
     
    #13
  14. DolphinsDan

    DolphinsDan Member

    Joined:
    Sep 17, 2013
    Messages:
    90
    Likes Received:
    6
    When do I get to see pictures?!?
     
    #14
  15. MacLemon

    MacLemon New Member

    Joined:
    Feb 16, 2015
    Messages:
    20
    Likes Received:
    19
    Thanks so much for all the input! I've combined all my replies into a single post to prevent a long monologue of mine.

    It's actually 1.4 years not months, but I agree that HGST drives are more reliable not only in Backblaze statistics but also in my personal experience. Considering to ditch the existing Seagate drives and go HGST only sounds like a viable way.

    Thanks for the kind words!
    I actually had to cut down to get below 10,000 characters. :) Would have been nice to have known about the limit beforehand which the posting form doesn't tell.

    It's correct, the A1SAM-2750F does have a small (1U) fan though the CPU lends itself to passive cooling. As I said, the motherboard is a compromise I'm forced to make and that fan is one of the downsides.

    As for mixing HDs of different brands:
    Most “4TB” HD models differ just slightly in the exact number of sectors they report to the OS. This also means, they actually differ a tiny bit in capacity. So if you start with one brand, wanting to extend/replace with another brand you may run into the issue that the replacement drives are a few sectors smaller preventing you from resilvering a RAID. I've not made the best experience in actually finding out the exact numbers reported by a mechanism to know whether I'll run into this issue. Manufacturers usually don't disclose this info, not even when explicitly requested. There is no guarantee that a line will not change slightly over its product cycle.
    The only way to be sure (imho) is to buy the drive and check the facts of that exact instance.

    I've also seen this issue with (cheap and not-so-cheap) external disk enclosures. Where one disk brand/model will work ok and another one can not even be partitioned at all since the drive firmware crashes due to some integer overflow in the firmware. Makes for nice back and forth email threads with manufacturers who blame everything but their own firmware even when proven.

    My tests with the similar C2758 were fine for my needs. YMMV of course.

    Thanks for suggesting the Nanoxia Deep Silence 6 (Rev. B) which looks quite interesting. It's said to fit “HPTX, E-ATX, XL-ATX, ATX, Micro-ATX, Mini-ITX” form factor motherboards. I'll see what Nanoxia has to say about fitting in the “proprietary form factor” SuperMicro A1SA7-2750F (21.08cm x 17.02cm / 8.3″ x 6.7″) board. The real issue is not the size itself which is basically µATX on one edge and mITX on the other edge, but the mounting holes which don't fit either standard. The mounting holes and no standard ATX power connector being the downsides of this model. I'd get by with just 2 × Gbit/s ethernet ports (instead of 4). Upsides being 16 SATA 3 connectors via LSI 2116 and passive cooling.

    This Atom CPU is totally different from past Atoms (like D525, etc.).

    Thanks for pointing out the RAM issue, I've updated my initial posting accordingly. (Hitting the 10k character limit again while doing so.)

    I guess my “day” job (as a sysadmin) shows. :)
    Thanks for the interesting motherboard suggestion. I'll look into it and see what CPU options arise with Socket 1150.

    PSU will depend on what motherboard I end up using. SuperMicro doesn't always have standard connectors. (Curse them for that!)

    That CPU is already EOL by intel.

    Not before I have actually started ordering parts. :) I'm still in the concept phase.
     
    #15
    Patriot likes this.
  16. Chuckleb

    Chuckleb Moderator

    Joined:
    Mar 5, 2013
    Messages:
    1,017
    Likes Received:
    330
    If you go with a standard ATX-sized MB from Supermicro, I've found that they all have the 24 pin power and 8 pin EPS per CPU. So if you're using 1 CPU, you need 1 x 8pin EPS.

    The thing about HDDs being different sizes, yes.. that is annoying. Most hardware RAID cards do rounding to get past this problem. The LSI cards do this rounding so that you can mix/match different vendors. You can do the same with software raid just by not using the drive to the full potential.
     
    #16
  17. DolphinsDan

    DolphinsDan Member

    Joined:
    Sep 17, 2013
    Messages:
    90
    Likes Received:
    6
    OK I will grab some tea and wait then :p I can't wait to see this one in action.
     
    #17
  18. Patriot

    Patriot Moderator

    Joined:
    Apr 18, 2011
    Messages:
    1,288
    Likes Received:
    662
    #18
    Last edited: Feb 19, 2015
  19. BlueLineSwinger

    BlueLineSwinger Active Member

    Joined:
    Mar 11, 2013
    Messages:
    149
    Likes Received:
    54
    The Xeon 'L' parts (as well as the Core CPU 'S'/'T' parts) only have a lower TDP because they've had their top-end limited. At idle they're really no different than the mainstream CPUs. And at load they're less effective (i.e., slower). Also, they're often more expensive.

    Intel's Low-TDP variants are intended for applications where thermal performance is limited, such as small/tight cases with little airflow. In a regular case they're wasted potential.
     
    #19
  20. NeverDie

    NeverDie Active Member

    Joined:
    Jan 28, 2015
    Messages:
    299
    Likes Received:
    26
    Hold that thought for a moment, because I really do want to get to the bottom of this once and for all. When I compare e3-1220L v3 (ARK | Intel Xeon Processor E3-1220L v3 (4M Cache, 1.10 GHz) ) with e3-1220 V3 (ARK | Intel Xeon Processor E3-1220 v3 (8M Cache, 3.10 GHz) they look completely different. The e3-1220Lv3 (13 watts TDP) has half the cores and 1/3 the base frequency of the e3-1220v3 (80 watts TDP). Yet you're saying that at idle they're really no different? At least on the face of it, that just doesn't sound right.

    @MacLemon: v2 and v3 are still "launch phase." v1 is EOL, though I'm not sure why that alone would matter.
     
    #20
    Last edited: Feb 18, 2015
    Patriot likes this.
Similar Threads: [Feedback sought]
Forum Title Date
DIY Server and Workstation Builds [Feedback wanted] Avoton A1SAM: 10GB NIC or SAS Controller in x4 Slot? Jul 31, 2015
DIY Server and Workstation Builds [Feedback wanted] Low-budget Mini-ITX build May 19, 2015

Share This Page