Starting Planning for DIY Home Server NAS

[XplodingData]

I have been running 4x4TB hard drives in my desktop, shared via windows, to play media on my TV on a RasPi running LibreElec. I also have a pair of external USB drives, and a stack of gopro media, photos, etc.

It's time to get a proper NAS up and running with some decent redundancy. ZFS2 seems good.

I think I've settled on FreeNAS (I'm running it to play around right now on some basic consumer hardware w/32gb ddr3 ram and it seems to work well/easily)

I will be buying a house and moving soon so I'd like to get myself setup with a little rack cabinet to include some basic networking gear as well as the NAS I build. I'd like to keep it clean and tidy.

Power consumption is something to be considered.

Goals:
- Run FreeNAS for the home network general data backup, media sharing
- Possibly transcoding of media library to a common standard file type
- 10GBe Fibfer link to main desktop PC (already have the hardware for direct link, probably look at a SFP+ switch down the road)
- Possibly mess about with some VMs (I'm curious, never played with them before - would just like it to be an option)
- Decent lifespan for hardware. I'm kind of cheap, but I would rather have hardware that's powerful enough to last me a while.
- Reasonable noise levels. If possibly I'd be happy to have active cooling on the CPU so I can use some nice quiet Noctua (or similar) fans.

Here's what I think I'm looking for:

- 35" deep 15U Rack Cabinet (or 24U if possible)
- <35" deep rackmount chassis w/ 16+ 3.5" Hot Swap Bays & SAS/SATA Backplane
- Mobo/CPU Combo capable of supporting 64gb ECC Ram (or more would be ideal), plus 10GBe SFP+ PCIe Card.
- PCI-e to NVMe Adapter card + NVMe drive (for cache)

If anyone has any suggestions for hardware or infrastructure, I'm all ears.

What I've sort of found poking around for a cabinets:
https://www.amazon.com/Server-Cabin...r_1_4?keywords=sysracks&qid=1580966799&sr=8-4

Found this guy for a chassis, but it's only 12 drives... would prefer 16+ so I've got some room to grow.
https://www.amazon.ca/Rosewill-Rackmount-Computer-Pre-Installed-RSV-L4412/dp/B00N9CXGSO

Appreciate any insight/help you can throw my way. Thanks!

 

[IamSpartacus]

My two cents:

FreeNAS is a great solution for storage, however in my experience and in discussion with others who run it, it's not a great solution for media serving. I'm not sure which media serving software you plan to use but FreeNAS does not support hardware acceleration (either iGPU or add-on GPU). That's kind of a big deal as HW Acceleration has gotten really good and takes a HUGE load of the CPU. Now if you are only going to be serving media locally at home and only 1-2 transcodes max EVER, then you are probably fine without it. Regardless, running services in Jails as opposed to something like docker containers in Linux just isn't nearly as flexible.

 

[itronin]

Obviously I've embedded a lot of opinions and what I've personally done flavors my post. What I've offered is by no means the only way or the right way or even the best way. Really its food for thought and hopefully ideas that open new avenues to look at in your journey.

Comments:
Building 1 server is a gateway drug. You will soon find you want a second and then a third... unless you have the discipline of @IamSpartacus

General:
Are you only looking at new or used is fine?

Rack:
Do you need/want fully enclosed, skins, doors etc?
Used:
Deals can be found on the Bay and Craigslist (or equivalent)
Check eBay (filter by distance) - If you can go 42U there are usually lots of options with doors and skins near major metro areas. HH - 24U are typically much more difficult to source.

New:
I have a sysracks 26U "U-Frame" rack. It is actually about 29U all said and done. I didnt' care about a door or skins. It works well. Instructions were lacking. My basement is ceiling limited so I could not use a 42U rack or I would have gone used.

Server:
New Chassis. The folks over at serverbuilds.net love the Rosewill chassis's. Std. ATX PSU, full height cards. I think its a bit limiting with 12 H/S drives. Everything is more or less std PC inside the chassis. Can be very customized for fans (sound, speed, etc. etc.)
Used Chassis. Supermicro 4U or 3U. Can be found with or without server guts.
Supermicro is really an ecosystem. Once you understand how the pieces/parts work and fit together it makes sense and there is a lot of cross chassis/system compatibility for different bits. Using non SM motherbaords in SM chassis can be done, done well, not that hard, but there is some effort (Front panel connector adapter/breakout for example)

with SM
Get or replace -R PSU with -SQ PSU (check compatibility first). 501-R psu is supposed to be quiet. 920-SQ are quiet.
3U and 4U chassis will accept FH cards.
4U - lots of eBay deals on CSE-846 chassis with SAS2 expanders, included ~350-500 shipped without rails.
3U - Compellent CT-SC030 can be found 150-200 shipped. you have to e-cycle the guts, and acquire cables to connect the backplane. The backplane is a "TQ" or direct attached backplane.
3U - CSE-836 - same kinda thing as the 846 - just 3U - these seem a tad more expensive than the 846's right now.
SAS2 expanders are great in the supermicro chassis and you don't burn a card slot on the motherboard.
SM TQ backplanes tend to be more flexible (and a bit cheaper to outfit, though more time consuming to cable up and work on).
SM rails - awesome, "snap-fit" style.
SM generic rails - cheaper, and if you won't be workign on the system in the rack or taking it out much then these are fine too. Personally I don't like them so much - but that's me.
SM PSU - good SM PSU's tend to be less expensive than good ATX PC (platinum level) PSU's. YMMV though.

Server Guts (I have experience with intel but not AMD) :
Used 1155 or 1151 - I think you may find it limited for number of PCIE lanes (thinking growth here), but tend to be power friendly. 1155 are pretty cheap and also popular over at serverbuilds.net
Used E5-26xxv1/v2 - enough pcie lanes with single or dual proc. DDR3 ECC memory is cheapest. cpu's are pretty cheap unless you try and get exotic. think 10 cores, 20 threads ~100.00. 8c/16T ~ 60.00
Used E5-26xxv3 - enough pcie lanes with single or dual proc. DDR4 ECC memory is coming down in price but still more than DDR3. v3/cpu's seem to be only slightly more expensive than v1/v2
HBA's - supporting 16 drives means an HBA at some point. Lots of $20 SAS 2 "IT Mode" HBA's out there. if you are comfortable with flashing firmware that it is. a little more expesnive if you aren't.

Network:
Do you need/want POE in your new House?
Are you putting in structured cabling?
Remember what I said about building a server can be a gateway drug? You may find you want 3 SFP+ sooner than expected and direct connection may become a PITA.
Take a look at the ever growing ICX Thread. Read the first few pages to see if that works for you. You probably don't need to read all 3K posts.
That said nothing wrong with direct connection.

Sparing:
If it is within your budget. Buy a shelf spare for common and important parts, SFP+ transceivers or HBA's as an example, for example: if a Dell PERC H310 is $18.00, buy two, flash 'em both to IT mode and put one on the shelf. Power Supply, again if within your budget, buy a spare PSU unless you have a uCenter or BB store, or AMZ prime and are okay with a day or three of down-time. NB: redundant PSU's give you a bit more breathing room (SM chassis, - can you tell I'm a fan?)

Used Part Selection - If it were me, I'd select parts that seem to be pretty easily available in qty on eBay or craigslist etc. so if I needed a spare I can go to the "online warehouse" and quickly get a replacement.

Use Cases - Its good you are thinking about these. Have you considered them along a timeline? Now, 6 months, 12 months, 18 months? Pick choices that will let you grow. I think this is even more important if you are planning a single AIO server...

Software:

Are you thinking of running a hypervisor and say FreeNAS as a guest? or are you planning to use Jails/VM's underneath FreeNAS?
If you are planning a hypervisor, which one?

Lastly: Have FUN with this project!

 

[XplodingData]

Wow ok thanks that's a ton of info. I'll have to dig a little deeper after work today.

IamSpartacus - my "serving media" is really just file shares. I'm not currently interested in streaming to devices other than my laptop/desktop and LibreElec Pi. So no on-the-fly transcoding. My transcoding would be background for long term conversion/storage, and could be done on my PC instead if needed.

ITronin - I will respond more thoroughly later but here's the quick reply.

I have run a 42U rack with diy servers in the past and feature creep is my M.O. lol. But this time around I'd like to keep it to about counter height so I can put the rack in my Office/Den if needed. I'd like the sides/enclosure to help baffle sound keep the cleanliness.

I'm happy to buy used products, but I'm thoroughly confused about the various chassis options. I have been surfing eBay and the posts here but I'm not quite there yet in terms of understanding the pieces. Please feel free to toss Me any links you know of.

10GBe long term plan is a SFP+ Switch with gigabit wired connections to a few computers/APs. I WILL be adding POE for cameras and the like down the road. This is the reason I'd like to leave a little growth room in my cabinet.

Spare Parts I'm down with. I airway have a spare pair of transceivers, and some extra cables. Now that I know the parts work as expected, I'll gather an extra card or two, same with my HBAs.

I have 6x Ironwolf, 4x WD Red that have been stress tested and are ready to go into production now. I'm watching for more sales to expand and grab spares.

Now the thought occurs to me that I'll probably like to run a video recording NVR setup as well... So whether that's in a jail or .... ?

 

[XplodingData]

K - got slammed at work, and then bought a new house soooo got a little distracted for a bit there.
But I'm back and pumped to get going on my build again.

with SM
Get or replace -R PSU with -SQ PSU (check compatibility first). 501-R psu is supposed to be quiet. 920-SQ are quiet.
3U and 4U chassis will accept FH cards.
4U - lots of eBay deals on CSE-846 chassis with SAS2 expanders, included ~350-500 shipped without rails.
3U - Compellent CT-SC030 can be found 150-200 shipped. you have to e-cycle the guts, and acquire cables to connect the backplane. The backplane is a "TQ" or direct attached backplane.
3U - CSE-836 - same kinda thing as the 846 - just 3U - these seem a tad more expensive than the 846's right now.
SAS2 expanders are great in the supermicro chassis and you don't burn a card slot on the motherboard.
SM TQ backplanes tend to be more flexible (and a bit cheaper to outfit, though more time consuming to cable up and work on).
SM rails - awesome, "snap-fit" style.
SM generic rails - cheaper, and if you won't be workign on the system in the rack or taking it out much then these are fine too. Personally I don't like them so much - but that's me.
SM PSU - good SM PSU's tend to be less expensive than good ATX PC (platinum level) PSU's. YMMV though.

Server Guts (I have experience with intel but not AMD) :
Used 1155 or 1151 - I think you may find it limited for number of PCIE lanes (thinking growth here), but tend to be power friendly. 1155 are pretty cheap and also popular over at serverbuilds.net
Used E5-26xxv1/v2 - enough pcie lanes with single or dual proc. DDR3 ECC memory is cheapest. cpu's are pretty cheap unless you try and get exotic. think 10 cores, 20 threads ~100.00. 8c/16T ~ 60.00
Used E5-26xxv3 - enough pcie lanes with single or dual proc. DDR4 ECC memory is coming down in price but still more than DDR3. v3/cpu's seem to be only slightly more expensive than v1/v2
HBA's - supporting 16 drives means an HBA at some point. Lots of $20 SAS 2 "IT Mode" HBA's out there. if you are comfortable with flashing firmware that it is. a little more expesnive if you aren't.

I like the look of the SM Chassis you suggested....
Found this (Supermicro 846E16-R1200B BAREBONE 4U Server BPN-SAS2-846EL1 24x TRAYS W. Rails 818240931370 | eBay) on ebay and I think it looks like the right call?
I'm a bit confused about hooking up the hard drives... I only see a couple of cables in the picture for the case, so are the expanders built into the backplane?

This ebay kit seems to come with a pair of 1200W PSU, but I'd be interested in ordering the lower wattage units that are quieter....
I ordered a couple spare HBAs for a good price and unfortunately USPS lost the packages on the scheduled delivery day... sigh. I'll give them couple more days and then I'll have to search again.

If this is a good chassis - then I'm on to motherboard/cpu combos... Single Proc is probably fine for this build, unless I can get a good deal.
I'll start googling about again, but feel free to suggest any links or deals.

Thanks for the help!

 

[XplodingData]

Still digging... I found a cheaper version of the same chassis (I think??) that has no PSU. (Supermicro 4U BAREBONE Server BPN-SAS2-846EL1 24x Trays No P/S W/ Rails 672042069316 | eBay)
I went to Supermicro website and found the PWS-920P-SQ power supply, but when I use the "validation status" checker function it says it's not validated with the CSE-846E16-R1200B chassis.... Does it actully fit/work? How can I tell?

I found some 920P-SQ units on ebay for cheap cheap, which is awesome. I'd love to pull the trigger on the cheaper chassis, plus a pair of 920 PSUs, but need someone of higher knowledge to let me know I'm not going astray...

I'm also a little confused as to how you hook up the motherboard once installed, as these server power supplies are far different than the ATX units I'm used to... I'll get to googling that shortly as well...

 

[itronin]

my advice. spec everything hardware and software before you start buying. I know - where's the fun in that! There are tons of 846's on the bay. enough that there will be some before you have everything spec'ed out. Have you also considered putting a post here in the WTS/WTT/WTB to see if someone already has one?

Did you settle on a software platform? from your use case I think that unraid is a good choice. But if you are doing nvr sooner than later - do you kow what nvr platform you want to run? so many puzzle pieces.

there will come a point though that spec paralysis sets in so you gotta make a choice and buy some stuff.

With the servers you spec'ed the expander is in the backplane. SAS2-846-EL1 - I hope I get this right. SAS2 is obvious so 6Gbps SAS2 or SATA, Number is the chassis, EL and number indicates the number of expander chips on the backplane . Unless you really need dual ported drives EL1 is probably the right number, less power, less heat too.

hmmm power. odds are pretty good that with the sas2 backplane in that chassis you should not have an issue in that the pdb is up to rev. the power supplies plug into a pdb (power distribution bus) that handles the redundancy and also can feed telemetry back to the BMC on the mainboard (IPMI access). the pdb has the wiring harness: ATX v2 power, usually dual CPU power, plus molex for backplane etc. for noise reduction you can run the server on one psu.

pdneiman/theserverstore/met are very reputable. all your links are from them I think. and techyparts and some others are good too.

If you are looking for 24 bay LFF chassis, search supermicro 846, cse-846, sc846 and then also look at SM's website for the part numbers for complete systems (start with motherboards that will work and look in the section for that baord that talkes about complete systems) and search for those too.

someone, and I can't find the seller or item, had an 846, 2x920sq, sas2-846-el1, generic rails, and I think x9dri with e5-2630v2's, 32gb of memory for a good price... can't find it now. kinda beats having to part it out but maybe reduces the fun?

on your motherboard search, I'd plan ahead either if you are going to run virtual blue iris (windows vm with gpu pass through) and/or gpu transcoding for media server then you want at least one x16 slot, two might be better, they don't have to be data x16, x8 data in x16 physical will be just fine as I think its well understood now for transcoding you don't need the full x16 bandwidth. If you want single cpu the x9srl-f or x10srl-f has qty 2 x16 physical slots. FWIW its that HP 290 that sells for 110 or so on amazon has intel quicksync and can handle hardware transcoding... of course that means another "server".

when specing I like to map theslot I need too:

for example:
10gbe (probably dual sfp+) = 1 x8
gpu one = 1 x 16 physical though having another would not be a bad thing if you media transcode *and* nvr transcode in two separate vm's.
1 hba = 1x8 (or maybe x4 in physical x8 if all spinners)
how many nvme do you want? each will be a x4 unless you get exotic with a switch hmmm intel p3605;s were back on the bay around 180.00 for 1.6TB... pretty good value card with a lot of write endurance. let's say you need one = x4
if you want to run say ESXI then you might also want a hardware raid 1 card for boot drives, in that case an x4 in x8 is enough for a pair of SATA SSD's in raid 1. say 1 TB each? actually x4 2.0 would work fine for that.

way I add that up so far
motherboard needs to have
pcie 3.0
2x8
1x16 (x8 data mini x16 phys) though 2 x 16 might be nice too.
1x4
pcie 2.0
1x4

I think

x9 or x10 srl-f could work really well if you want to go single cpu...

I think

x9dri-f or x9drh-f would work well for dual cpu and have an x16 slot, I haven't really investigated the x10 dual cpu boards yet so I have no opinion
x9dri-f does have two x16 slots though.

more food for thought for ya? Probably too much rambling - sorry about that. its been a long day.

 

[XplodingData]

You're right about the speccing then buying, but unfortunately life is going to be silly busy for me the next few months, which will quickly turn into longer I'm sure. My drives are currently screwed to a strip of plastic out in the open so I need to get something started.

I'm also perfectly happy to read long rambly posts that are full of helpful information. Super thankful for the help, it had been tremendously useful.

I believe the 920SQ PSU will fit, they appear to be the same size and output pin config... Plus the chassis model seems to indicate the (newer?) 8484 PDU which lists it as compatible.... So I'll probably order and try them.

So my current focus goals are

Primary Use: File server
Secondary Use: media transcoding for file storage (not on the fly, just background batch work to standardize some files for storage)
Third Use: NVR (maybe... But I'm also willing to just use a basic NVR unit)
Power Usage: low as reasonably possible without spending a ton extra to get there (power is only about $0.11/KWh here)
Noise: as low as possible for in home installation


Expected Requirements:
16+ bays for 3.5" spinners (24 bay 846 case looks perfect)
920SQ PSU X2
Single CPU DDR3 mono w/64+ GB RAM.
10gbe SFP+ PCIe card (airway purchased)

A dual cpu sounds great, but really this is a basic server. I was happy using my desktop but I'd really like to get ZFS redundancy and I can't fit 12+ drives in my desktop...

That said, if you stumblr across any great deals, please let me know. I'm googling a bit in the evenings now but I'm going to have to start packing/moving soon sooooo free time is limited. Sigh.

 

[Marsh]

Noise: as low as possible for in home installation

The 24 drives SM chassis is opposite of low noise.
Bigger chassis is not always better.

Today, with 14TB drive, 8 drives goes a long way.

 

[nthu9280]

If you want put Rack in your Home office / Den, be very mindful of heat and noise.
I’d prefer to keep them away from living areas.
For example, I have one WS/Server in my home office and that sucker can put out some heat and noise when I run ML /DL tutorials not anything substantial. Drive noise is another factor.
SM743 tower, quiet fans, gtx 1060 6gb, X10SRA-F with E5-1620V3.
Other stuff is in my Garage 24u rack.

 

[XplodingData]

I'll clarify, for the next couple months it'll be in my place as I build it, then I'm moving into a house with garage, and it'll go into a rack in the garage area. I do plan to spend time out there so I don't want a noise machine, but I'm sure the 4U chassis will be ok for it. The heat will be welcome in the winter.

I have 10x 8tb drives now, and 4x4tb that currently house all my media/files, so I'm not really interested in buying larger drives just to get the count down. A larger case is just fine for me, plus the additional bays means I can hot swap drives in for backups and eventually create another pool down the road, or even just have spare trays on hand.

 

[XplodingData]

The 24 drives SM chassis is opposite of low noise.
Bigger chassis is not always better.

Today, with 14TB drive, 8 drives goes a long way.

Can I not swap the fans for reasonably quiet ones? And after googling I'm now under the impression that most of the noise is from the power supplies, which I'll try and swap out to the SQ variants.

 

[Marsh]

Can I not swap the fans for reasonably quiet ones?

You would need high static pressure fans to cool 24 drives in the Supermicro 4u chassis.
It really depends on your garage temperature in summer.

 

[XplodingData]

You would need high static pressure fans to cool 24 drives in the Supermicro 4u chassis.
It really depends on your garage temperature in summer.

It's definitely something I'll have to look into. I've got no issues with a little custom work to fit some larger fans as well if that's possible.

Worst case I have to power it off when I'm not home using the network until I get it in the garage. I don't expect garage temps to get very crazy in the summer. Maybe 30C on a really hot day.