Follow up, after running this build for a good 18 months now. Anyone interested what my experience is? Well, it goes something like this…
The case: (Nanoxia Deep Silence 1)
The whole thing is barely audible. It gets somewhat warm with the stock fans during the summer, which is still easily compensated for by turning the fans up and opening the case chimney as a last resort.
Dust is no issue given one cleans out the intake fans, which I do once every season. The inside has accumulated less dust than expected. I'll spare you the bottom dust filter since I didn't want to shut down the machine to get it out safely. You have to believe me when I say that it's still almost clean as shipped. Some minimally gross pictures follow.
The 8x drive cage, of which 6 are populated with HGST NAS 4TB drives are almost clean. HDs haven't topped 50°C ever which is in the top range of acceptable temperature for me.
I'm using the FreeNAS default scrub interval of 35 full earth rotations, plus an occasional second. Not a single repair was necessary with neither, the 6 drive HGST RAID-Z2 run from the HBA, the 3 drive Seagate 4TB RAID-Z connected to the onboard SATA ports nor the SATA DOM I use for FreeNAS itself.
The tiny CPU fan does its job just fine keeping the heat sink shiny and does so silently, especially through the heavily padded case.
The IBM M1015 HBA (an OEM LSI 9211-8i) works perfectly fine and delivers a solid 550MB/s of read performance. Absolutely no issues. I've flashed it to firmware revision 20 by now which matches what FreeNAS 9.10 expects.
The fanless power supply doesn't show any signs of whine or other noise. Still has plenty of power to spare for more drives I intend to add.
Ending up at the back of the case we see the exhaust fan happily spinning silently dissipating off of hot air from the case.
Usage:
I'm running over 20 Jails, plus an ever changing handful of VirtualBox based VMs.
The 32GB RAM seem to be plenty for what I do and provide enough ARC that I don't see any reason to add an SSD for L2ARC. The upper ARC amount topping around 28GB and the lower mark around 10GB.
Issue:
I can only recall a single issue that forced me to actively intervene. This was an update of Plex server that had gone awry. It was fixed manually by instantiating a new Plex jail and porting over the data.
Things I'd do differently building a similar box today:
When planning my capacity I've stupidly missed that ZFS is most happy when allocation stays under 80%. So when calculating your net capacity, don't forget to multiply the result by 0.8 to arrive at the capacity you can use at full performance.
FreeNAS and ZFS will continue to run just fine beyond that point, but ZFS will switch to space saving mode which will slightly hit performance.
Over the last year projects have emerged, that happened to generate a few TBs of data which I had not anticipated when I crafted the specs.
No matter how much storage you provision,
you'll eventually run out of space.
I'd strongly consider a Xeon-D based motherboard over the Atom C2750. More for the 128GB RAM limit and 10GE capabilities than CPU power. I'm still happy with the processing power my choice provides and I also get along quite well with my 32GB.
I'd do the Molex to SATA power cabling different, since I'm unnecessarily taking up Molex ports on my power supply which results in rather inelegant cabling. I don't care about the optics, this is not a gaming rig on display at a trade show after all. I do care for unobstructed airflow, effective cooling and silence though as well as easy maintenance. The existing, somewhat messy cabling is contradicting that.
So during upgrading my server I'll be changing the SATA power wiring to these. (I don't care about the colours, it's just what was available.)
Upgrading the box:
I'm in need of upgrading my storage pool. At the moment I'm running two separate zpools, one 6 * 4TB RAID-Z2 for storage and another 3 * 4TB RAID-Z pool for client backups.
I'm not yet sure how I'll be transmogrifying my data from the existing pools to the new pools though. I would have loved to just add the new 8-drive RAID-Z2 as a mirror to the current 6-drive RAID-Z2 and have ZFS work its magic to resolver the mirror, then remove the old pool from the mirror. From what I know about ZFS this is not possible so I'll have to make do with the FreeNAS replication function.
I'll be switching the main storage pool to an 8 * 6TB RAID-Z2 pool which means, I cannot do an in-place upgrade disk-by-disk since I'm changing the number of devices (from 6 to 8) in the pool.
I would have liked to have a downtime free option for data migration, even though a day or two of downtime is really not an issue for me in this case. Would be a different matter for a customer site though.
I've had good experience with the HGST NAS drives, so I'll be going with HGST 6TB NAS. Currently they go for shy over 200€ a piece excluding VAT.
I'll also be upgrading the backup pool to an 8 * 4TB RAID-Z2 zpool subsequently. (Mostly for local and remote clients.)
This also requires me to increase the number of SATA connections. Drive speed is not that critical, given that I access everything over paltry Gbit Ethernet, meaning I'll be doing 400MB/s at max. It's more like a maximum of 200MB/s in practice.
One solution would be to swap out the existing 8 channel SATA-III HBA for one with 16 channels, or insert another 8 channel HBA into the remaining PCIe slots.
A different solution would be to go for a SATA port multiplexer and drive both pools from the 8 channel HBA with a port multiplexer to the 16 drives.
I'm not sure if the latter solution is viable, it certainly would be cheaper though. Feedback and suggestions on this are much appreciated! (I'm not really familiar with add-on SATA port multipliers.)
Conclusion:
Overall FreeNAS has proven to be extremely reliable for me and I happily recommend it for similar use cases like the ones I've depicted here.
Future:
I'm currently working on server builds for customers and my own company at the moment. (Smaller Xeon-D and dual E5 FreeBSD servers as well as a dual E5 based storage server (FreeNAS 9.10 based) with around 72TB of raw capacity.) Hopefully I can let you in on some more hardware pr0n then.
Edits:
Eek, typos.