Suggestions For Setup

[AsinineLucky]

Hey Everyone, first time posting here, so apologies if this should be in a better location. I am trying to build up a couple servers to provide a base for my house as well as my parents. The ultimate goal is I would like to have them running FreeNAS for data storage (Media files and other documentation backup), as well as host PLEX and run VMs for various tasks and labs.

My initial thought from reading on the forum is to install ESXi on a SD card which will be mounted internally. From there, I can host my VMs on one drive (PLEX and FreeNAS for instance) and passthrough the other three disk drives directly to FreeNAS for RAID management. Initial thoughts on this?

To provide some hardware details:

I have two HP Proliant DL360e Gen8 servers with:
2x Intel Xeon E5-2450L 1.8GHz
48GB DDR3 RAM (24GB/processor)

Hard Drive wise I was looking at getting 4x 4TB WD RED (Unless the 6TB drives would be better recommended)

Since I would only have 3 usable drives for FreeNAS, I would essentially run a RAID 5 configuration for a 8TB total capacity. Happy to hear the recommendations and if there is a better way to setup the configuration.

 

[Spartacus]

So what are your current and/or future plans for this as well as expansion?
If you're set on vmware and you're running a host that can only hold 4 disks, I would just use hardware raid and make a single large datastore personally.

That said the passthrough and vmware is extra work, cost, licensing, and headache too.

If you want the flexibility to migrate to new hardware FreeNAS supports running virtual machines, why not use it's built in option and have FreeNAS be the OS directly? or even better run plex and anything else you can in a docker container?

A third option would be the hardware raid 5 and just install linux directly on the disks to share out via NFS/SMB.

Unless you're planning on migrating these disks to another host in the future, I would just go with hardware raid to reduce complexity.

 

[AsinineLucky]

My goal is to migrate my Plex server off of some aging hardware, as well as to run a few VMs. The VMs will consist of a lab environment for penetration testing and the like, as well as a windows box to host a few applications that I don't want to keep on my everyday computer. Ultimatly I just want to ensure that my data is secure. For what it is worth, theses servers also have ECC memory which will work great with the FreeNAS ZFS file system.

The NAS itself was wanted to host all of the movies and documents, as well as potentially to create a private cloud storage of sorts. Using NFS for sharing the storage back to ESXi if needed, or just as a share to the VMs and household devices.

I am not set on using ESXi, but I do have experience with it and thought I would leverage that. Docker will certainly be a learning curve for me to get is setup, but willing to try if that is the recommendation.

I can certainly do a hardware RAID, just was under the impression from reading posts that between FreeNAS doing a ZFS setup and the overall usability of a software RAID that was the path to take. Appreciate the feedback!

 

[Spartacus]

For what it is worth, theses servers also have ECC memory which will work great with the FreeNAS ZFS file system.

For what it is worth, they recommend ECC, but theres very little harm/risk of using regular memory for homelab/use scenarios.
That said the rdimm are much easier to get in large size dimm, its best if you get as much ram as possible if using Freenas for cache.

The NAS itself was wanted to host all of the movies and documents, as well as potentially to create a private cloud storage of sorts. Using NFS for sharing the storage back to ESXi if needed, or just as a share to the VMs and household devices.

I am not set on using ESXi, but I do have experience with it and thought I would leverage that. Docker will certainly be a learning curve for me to get is setup, but willing to try if that is the recommendation.

You're where I was at about 4 years ago, I was big on VMware and had just gotten my VCP and signed up for the VMUG to get licenses.
It worked for what I was doing and would for you too, but looking back it was really inefficient and has a yearly cost associated for the vmware license (unless you ahem...procure some licenses from work ). I moved off it about 2 years ago to unraid and haven't looked back.

Docker in freenas/unraid is pretty easy to learn there are alot of great guides out there, especially for setting up Plex, it has an extremely low overhead by running individual apps you need in containers rather than setting up separate VMs for them, my main NAS doesn't even have VMs anymore just a bunch of docker containers and its using about 1/3 the resources it was previously.

I can certainly do a hardware RAID, just was under the impression from reading posts that between FreeNAS doing a ZFS setup and the overall usability of a software RAID that was the path to take. Appreciate the feedback!

Thats what I asked what your future expansion storage needs and otherwise were, theres a couple of key benefits to the software raid.
1) Hardware agnostic so you can move the disks to any other hardware, start it up and with minor tweaks have your system up and running without compatibility or brand locked in requirements for re-initializing the array.
2) Ability to add additional disks/pools ~ if this part is important to you I would actually recommend considering unraid, its not as performant as freenas/ZFS on linux however instead of having to match pool disk sizes you can add individual drives to expand your storage and convert/change the redundancy live without having to recreate the array, it will live convert it.

 

[AsinineLucky]

Docker in freenas/unraid is pretty easy to learn there are alot of great guides out there, especially for setting up Plex, it has an extremely low overhead by running individual apps you need in containers rather than setting up separate VMs for them, my main NAS doesn't even have VMs anymore just a bunch of docker containers and its using about 1/3 the resources it was previously.

I will start reading up on Docker then, and maybe go ahead and install FreeNAS to start playing around with the setup. How well would this work for running an actual Windows OS? For instance, one of these boxes was going to be used by my father to run applications he does not want taking up room on his laptop he travels with for business. I was also going to virtualize pfSense on his since I have a dedicated server running in my setup already. That will need to be hosted as well.

1) Hardware agnostic so you can move the disks to any other hardware, start it up and with minor tweaks have your system up and running without compatibility or brand locked in requirements for re-initializing the array.
2) Ability to add additional disks/pools ~ if this part is important to you I would actually recommend considering unraid, its not as performant as freenas/ZFS on linux however instead of having to match pool disk sizes you can add individual drives to expand your storage and convert/change the redundancy live without having to recreate the array, it will live convert it.

That makes sense. I would like the ability to upgrade the storage space down the road as they fill up. Plus as I am converting older hardware for this purpose I feel having the flexibility a software RAID has may be more in line from what you mentioned.

Thanks again for the feedback, already giving me more to think about.

 

[Spartacus]

I will start reading up on Docker then, and maybe go ahead and install FreeNAS to start playing around with the setup. How well would this work for running an actual Windows OS? For instance, one of these boxes was going to be used by my father to run applications he does not want taking up room on his laptop he travels with for business. I was also going to virtualize pfSense on his since I have a dedicated server running in my setup already. That will need to be hosted as well.

It kinda depends, so I'm running about a half dozen docker containers and two VMs on my unraid currently.
One is just a straight windows box that I use as a Dropbox sync, I'm too cheap to pay for multiple accounts so I have it running windows and shared dropbox to my wifes computer so she can see the multiple hundreds of gigs of the family photos/videos and interact with them (only has 500gb on her laptop so she cant sync it all). I just RDP into it when I need to mess with it (theres also a VNC console option with unraid).

In the instance of your dad, you could definitely setup a windows VM he could RDP into that runs more resource hungry programs, or he doesnt have space for etc. it would be only accessible at home without some kind of VPN though.

You can virtualize pfSense but personally I recommend leaving it as it's own appliance otherwise if you have to do maintenance on your NAS you have to take down your internet too. I've basically gone full ubiquiti for my networking needs at this point I got tired of dealing with custom and just wanted something turn key thats one click to update/patch FW.

 

[AsinineLucky]

Appreciate the tips! Getting a sample put together while my new drives come in. Since I was going to have the 4 x 4TB drives, and another drive internally for hosting TrueNAS, which RAID configuration is suggested for the 4 drives? I was initially thinking RAIDz1 since I get the most usable space, but that gives me one point of failure right?

 

[Spartacus]

only do z1 if you dont care about the data
z2 or in your case a mirror would make the most sense (z2 works best with 4, 6, or 10 drives)

 

[Sean Ho]

If you're set on zfs rather than a unionfs style (e.g., Unraid or snapraid+mergerfs), then depending on future expansion plans, you might also consider pool of mirrors (RAID10): you lose the ability to recover from failure of *any* two drives, but you can still recover from multi-drive failure as long as one drive from each mirror pair is still intact. Recovery only thrashes the mirror drive, as opposed to all the drives in the vdev. And most importantly, you can easily expand by adding drives in matched pairs (e.g., add a 2x 6TB mirror vdev to the pool).

I'd also recommend a cheap SSD (SATA or NVMe) for VM/docker images. You can back it up nightly to the zfs array, or get a second one for a mirror pair (zfs, btrfs raid1, or mdadm raid1).

If you find transcoding performance sluggish, consider adding a cheap SFF box with Intel QSV, 7th gen or better CPU. Mount the media from the NAS over NFS.