Looking to upgrade to AMD Home Server

[gunndykol]

Currently I have a home server running dual Xeon E5-2670 8 core processors 128GB of DDR3-1600 ECC memory that I run ESXi 6.0 on. It is in a Rosewill 4u server chassis with 12 3.5" drive bays full of HDDs.

I currently run a handful of full time VM hosts:
- FreeNAS Server
- Plex Server
- Windows 10 Blue Iris "server" for 5 home security cameras
- MS Active Directory domain controller
- Dedicated Linux Torrent/Sonaar/Radarr VM

I work in IT (desktop side) and am constantly spinning up new VMs to use as test labs to learn and experiment with Citrix, Linux, and all sorts of other things.

My current system has enough CPU cores and memory available to do pretty much anything I've wanted to do; but isn't the most power efficient solution out there. I have been considering replacing it with a new AMD Threadripper or Epyc homeserver build. I have gotten my hands on 6 sticks of DDR4-2666 ECC ram and am trying to figure out a good budget option for CPU and MB.

I am considering grabbing a Epyc 7551p from ebay and a budget sp3 motherboard; but wanted to post on here if anyone had any ideas that might be better in the same price range ($500 to $600 total for MB and memory). Also wondering if anyone has had any experiences (good or bad) with EPYC and ESXI. I'm a desktop guy and haven't really done much work in server space and I've read some stuff about memory configs and NUMA settings (which I really don't understand) but and trying to learn more.

If anyone has any advice or guidance, it would be greatly appreciated.

 

[Wasmachineman_NL]

Why not wait a few months in the hopes Zen 2 Epycs become cheaper?

 

[TLN]

Seems like overkill, going from legacy DDR3 system to latest threadripper.
I'd stick with Xeon E5-2600v3/v4 or newer scalable. Sure, not as fast as maxxed out threadripper, but we're talking 2-4x your system (im talking abot high-core count processors). With efficient psu it will sim energy and can be had pretty silent.

 

[zer0sum]

Have you actually measured the power usage of your current server at idle and under load?

That might give you an idea of whether a newer system might be different.

 

[msg7086]

E5v1 is pretty efficient already. On idle they reach 60w per socket.

https://www.servethehome.com/dual-intel-xeon-e52690-update-power-consumption-foldinghome/

STH measured it 120w idle on dual socket server.

https://www.servethehome.com/amd-epyc-7551p-benchmarks-and-review-a-32-core-value-monster/4/

STH measured EPYC 7551p at 81w idle.

If you want to lower the power consumption, you can simply remove one CPU, assuming the rest is still enough for your daily work. That would push it to the same power level as the EPYC.

Alternatively, sell the pair of 2670, and get a 2690v2, assuming your server works with E5v2 series.

However even if it uses 50w more, that's only $50-100 extra on the power bill per year. Upgrading to EPYC may not pay the difference back in the near future.

 

[gunndykol]

Why not wait a few months in the hopes Zen 2 Epycs become cheaper?

That's a never-ending cycle, as you can always wait for the next tier to become cheaper. Also, I don't really see anything that Zen2 brings me over Zen1 right now other than slightly higher clock speeds, which for a home server and my uses probably wouldn't be seen in real life workloads. I would definitely look to try and buy a Zen2 capable MB just in case I ever feel the need to move to 2nd gen in the future.

 

[Wasmachineman_NL]

That's a never-ending cycle, as you can always wait for the next tier to become cheaper. Also, I don't really see anything that Zen2 brings me over Zen1 right now other than slightly higher clock speeds, which for a home server and my uses probably wouldn't be seen in real life workloads. I would definitely look to try and buy a Zen2 capable MB just in case I ever feel the need to move to 2nd gen in the future.

Zen 2/Rome has the IOD and less screwing around with NUMA.

 

[gunndykol]

E5v1 is pretty efficient already. On idle they reach 60w per socket.

https://www.servethehome.com/dual-intel-xeon-e52690-update-power-consumption-foldinghome/

STH measured it 120w idle on dual socket server.

https://www.servethehome.com/amd-epyc-7551p-benchmarks-and-review-a-32-core-value-monster/4/

STH measured EPYC 7551p at 81w idle.

If you want to lower the power consumption, you can simply remove one CPU, assuming the rest is still enough for your daily work. That would push it to the same power level as the EPYC.

Alternatively, sell the pair of 2670, and get a 2690v2, assuming your server works with E5v2 series.

However even if it uses 50w more, that's only $50-100 extra on the power bill per year. Upgrading to EPYC may not pay the difference back in the near future.

I haven't actually measured my server draw on average from the wall, and probably should do get a baseline idea. While the 7551p may not be 1/2 the power draw, it's still a measurable difference along with doubling the # of available cores and threads along with advancements in core clocks and IPC seems like an overall win.

Part of this "upgrade" is also a chance to move away from eSXI and try out the new TrueNAS Scale.

My current server build: https://pcpartpicker.com/user/Gunndykol/saved/sJ9frH

I also potentially have a friend willing to pay me for my current Xeon base components (CPUs, MB, and RAM) to build their own first home server/lab, so I will recoup some of the $ from there as well.

As you can probably tell, another part of this upgrade is just the idea of moving to something a little more modern and newer. As it's more for home learning and tinkering and not for business, I'm not overly concerned with whether the upgrade pays for itself in energy costs quickly; as it also offers me the chance to expand my number of VMs for home lab/self training.

You do raise some good points for me to consider however.

 

[gunndykol]

Zen 2/Rome has the IOD and less screwing around with NUMA.

I admit that I have read about Zen and NUMA, but I really don't know enough about it (and probably should learn more). I've never heard IOD before and don't know what that is.

 

[gunndykol]

Seems like overkill, going from legacy DDR3 system to latest threadripper.
I'd stick with Xeon E5-2600v3/v4 or newer scalable. Sure, not as fast as maxxed out threadripper, but we're talking 2-4x your system (im talking abot high-core count processors). With efficient psu it will sim energy and can be had pretty silent.

v3/v4 is not an option for me as my current MB only supports up to v2 options. If I have to buy a new MB as well, then I figure I might as well look to switch to EPYC as they offer better options if I want to go over 10 cores/20 threads in a single CPU for lower power draw. Also, the number of PCI lanes available on EPYC is insane for PCI expansion and storage.

 

[Wasmachineman_NL]

I admit that I have read about Zen and NUMA, but I really don't know enough about it (and probably should learn more). I've never heard IOD before and don't know what that is.

IO-die.

 

[TLN]

v3/v4 is not an option for me as my current MB only supports up to v2 options. If I have to buy a new MB as well, then I figure I might as well look to switch to EPYC as they offer better options if I want to go over 10 cores/20 threads in a single CPU for lower power draw. Also, the number of PCI lanes available on EPYC is insane for PCI expansion and storage.

oh, I totally understand that epyc will be more powerful. It also will be more expensive for sure.
You can easily get 14-18 cores in v3/v4 platforms, but if you need more epyc is the way to go.

 

[gunndykol]

oh, I totally understand that epyc will be more powerful. It also will be more expensive for sure.
You can easily get 14-18 cores in v3/v4 platforms, but if you need more epyc is the way to go.

To my knowledge I can't go to v3/v4 platform without upgrading my existing motherboard. Unless I'm missing something the 602 chipset 8s v1/v2 only.

 

[TLN]

To my knowledge I can't go to v3/v4 platform without upgrading my existing motherboard. Unless I'm missing something the 602 chipset 8s v1/v2 only.

That's almost correct. I recall seeing some v3/v4 working with DDR3 on chineese motherboard. Haven't tried myself, so cannot recommend it.

I was saying that high-core v3/v4 will be close to low-core epyc, and both will be significant performance upgrade over existing system. Both are DDR4, so no difference there (except of cheap 2133 memory for v3).

 

[gunndykol]

After a little more looking around on ebay, I decided to just pick up two e5-2697 v2 CPUs and 128GB more DDR3 to max out my existing server for a little over $300. 24 Cores / 48 Threads should be enough to finish out all the VMs I want for my test lab and doubling the memory will allow me to up the amount on my existing VMs to help performance. Still debating on if I wanna switch over to TrueNAS Scale though instead of ESXi as the hypervisor.

Switching to TrueNAS Scale gives my NAS access to all the memory and running the Plex app should give it access to all CPU cores for transcoding vs running TrueNAS core and Plex inside two separate ESXi VMs to hopefully maximize that performance; unless I'm missing something. That then leaves me running my test labs and Active Directory from VMs.

 

[zer0sum]

After a little more looking around on ebay, I decided to just pick up two e5-2697 v2 CPUs and 128GB more DDR3 to max out my existing server for a little over $300. 24 Cores / 48 Threads should be enough to finish out all the VMs I want for my test lab and doubling the memory will allow me to up the amount on my existing VMs to help performance. Still debating on if I wanna switch over to TrueNAS Scale though instead of ESXi as the hypervisor.

Switching to TrueNAS Scale gives my NAS access to all the memory and running the Plex app should give it access to all CPU cores for transcoding vs running TrueNAS core and Plex inside two separate ESXi VMs to hopefully maximize that performance; unless I'm missing something. That then leaves me running my test labs and Active Directory from VMs.

If you have a spare PCIe slot, you should look at a cheap Nvidia P400 (or similar) for transcoding