Looking for Xeon-D / Atom home server recommendation

[jcl333]

Hello all,

A while back I was following some of these great Xeon-D setups with like 16-cores and low power consumption, but then I noticed some newer Atom's come out that looked as though they maybe fit in a similar segment. I think the Xeon-D even has RAS. I got busy with other things and was out of it for awhile. So, I am hoping some people here who are following it closely could give me a recommendation.

Here is my first use case:
* A flexible virtual host to support 5-6 VMs at any one point in time (ESXi 6.5, etc.)
* A virtual pfsense box, and I will also be looking at other similar solutions like Sophos UTM, Untangle, possibly together using transparent proxy / bridge mode. Trying to find the best solution for protecting my home network of about 12 or so computers.
* My Internet connection is 1Gig FiOS fiber

So, I could get a box sized just to be a network gateway appliance like that, but it seems to me that it would be more efficient to get one box that can do multiple use cases vs. separate boxes, but let me know if people agree. I work as a systems engineer so new things come along that I may want to have a place to try.

Here is my second use case:
* I have a 10-bay QNAP box with about 16TB of space
* I would like to have a 2nd NAS to backup the QNAP, maybe using FreeNAS
* Basically startup a VM, replicate NAS, shut down as a weekly job
* So this could be a VM with an HBA passed through to the guest to give direct access for ZFS (or similar)
* The server I use could either have bays for the hard disks in it, or just be say a small box with a PCI-E slot, and use external SAS on the HBA to another box with disks in it.

Here is the type of hardware I was thinking:
* Small form factor chassis with 4 or so bays (probably quieter / more flexible than 1U box)
* Motherboard probably SuperMicro with support for SR-IOV
* Xeon-D 12-16 cores, or similar
* 32-64GB ECC RAM (start with 2x16 and leave slots open for future)
* Multiple 1Gig Intel NICs, at least two preferably 4... 10Gig is tempting but may just add cost
* NVMe SSD for VM files, DOM, USB, or SD for ESXi

So, let me know if anyone has a good recommendation that might be a good fit.

Thanks

-JCL

 

[Evan]

I would say too many cores for the ram, unless what your doing is especially cpu intensive the high clocked d-1541 could be a better fit at a lower price. Having said that I am sure you know your workload better.
If your really after more than 8 cores the maybe is e5 v4 or scalable Xeon time.

 

[jcl333]

I would say too many cores for the ram, unless what your doing is especially cpu intensive the high clocked d-1541 could be a better fit at a lower price. Having said that I am sure you know your workload better.
If your really after more than 8 cores the maybe is e5 v4 or scalable Xeon time.

If I take out FreeNAS and fill that role some other way, then I would say sure, and that is definitely a consideration.

I have not used FreeNAS before, wondering if things like de-duplication and compression would need some horsepower.

But I might just keep it simple and get another QNAP to backup the first...

-JCL

 

[K D]

When you say 5-6 vms, what you plan to run on them matter.

Here's an example :

1. Freenas VM with hba pass through
2. Plex
3. Sab/sonarr etc
4. Unifi controller
5. Win 12r2 AD/dns/dhcp
6. Pihole

All this fit in a flex atx xeon D 1518 with 16GB ram with space to spare. Ran for 2 months without any glitches as my main home server. Only issues were in playing more than one transcodes stream in parallel in plex and 4k transcodkng in plex.


I ended up moving to an E3-1275 v6 for my home server for higher clock speed and intel quick sync. Also in my chassis (unas 210a) a flex atx or mini itx board made for poor airflow across the drives and I needed a bigger motherboard.


Also note that if you go the Xeon D route, the mini itx boards have only a single pciex16 slot.

 

[jcl333]

When you say 5-6 vms, what you plan to run on them matter.

Here's an example :

1. Freenas VM with hba pass through
2. Plex
3. Sab/sonarr etc
4. Unifi controller
5. Win 12r2 AD/dns/dhcp
6. Pihole

All this fit in a flex atx xeon D 1518 with 16GB ram with space to spare. Ran for 2 months without any glitches as my main home server. Only issues were in playing more than one transcodes stream in parallel in plex and 4k transcodkng in plex.


I ended up moving to an E3-1275 v6 for my home server for higher clock speed and intel quick sync. Also in my chassis (unas 210a) a flex atx or mini itx board made for poor airflow across the drives and I needed a bigger motherboard.


Also note that if you go the Xeon D route, the mini itx boards have only a single pciex16 slot.

So, basically what you are saying is I am very likely greatly over estimating my need. This is actually helpful, gives me some perspective of what this hardware is really capable of.

There are a few details that are not in your description that might matter, such as what you task the FreeNAS with, and the specs on your storage. You could easily use that whole box with RAM for FreeNAS alone, in some scenarios.

I have heard of Plex but don't know much about it, not sure what Sab/sonarr and Pihole are. Not surprised about transcoding shortcomings, honestly it would not have dawned on me to even try that on a server like that.

Never really felt the need to run AD at home, because I do so much of that at work.... and BTW I would never run just one DC if that has anything you care about.

I have a Unifi hotspot and have just been using the java controller on my PC on an as-needed basis when I need to make a change, is this some kind of virtual appliance you are using?

The single PCIe slot should be OK.... I only see adding maybe an HBA, since most of these boards are available with plenty of network connectivity, not sure what else you would use it for.

Thank you for the info

-JCL

 

[K D]

FreeNAS was hosting a zfs pool with 8 8TB drives providing SMB storage for media, security cam recordings and general backups from my workstation. And a second SSD pool with 2 drives providing nfs storage for VMs.

My whole home network is on Unifi. I use the USG, and about 5 Unifi switches, 3 APs and their cameras. I run the controller in a debian VM.

Note that what I described above is the "home server". I agree freenas can consume use more RAM. But for the use I described I don't see a point in more RAM. I'm able to get sustained transfer rates of ~260MB/s which is more than gigabit ethernet. I had given it 8GB because that's the minimum requirement. After moving the the E3, I no longer use Freenas and use omnios/napp-it. The same pool gives the same performance using omnios/napp-it and 6GB of RAM.

And overall power utilization did not go up by much after moving to the E3.

 

[Joel]

Everything I have heard about dedupe in ZFS is that it uses a metric err... truckload of ram. Not sure how much because I've never tested. I suppose a good way to dip toes in that water is to enable dedupe in a small zvol and see what happens before scaling.

 

[PigLover]

Dedupe has very limited value in a home/LAN use case. Just turn it off and ZFS stops being a ram hog.

ZFS compression requires very little overhead.

Sent from my VS996 using Tapatalk

 

[CookiesLikeWhoa]

Everything I have heard about dedupe in ZFS is that it uses a metric err... truckload of ram. Not sure how much because I've never tested. I suppose a good way to dip toes in that water is to enable dedupe in a small zvol and see what happens before scaling.

FreeNAS quotes 5GBs of RAM per 1TB of usable space for dedupe.

Agreed not worth it for the home network.

As others have said that amount of RAM is light for that many cores. My personal rule of thumb is 32GBs of RAM for ever 4 cores. This is just my general feeling. If you're doing something CPU intensive (Like running BlueIris) then you can get away with a lot less RAM and more Cores since that program eats CPUs for breakfast. Flip side if you're doing a lot of lightweight VMs then you could end up needing even more RAM.