EPYC! 7452 AIO Home Storage/Media/Virtual Desktop Server

[IamSpartacus]

For all you regulars on these forums, you know I change hardware like my wife changes channels (wait...that's an inside joke). Just see my PMS 4.0/5.0/6.0/7.0 Build Thread for you newbies.

Anyway, when I built my last iteration of the home server I was intending to next build myself and my wife a PC. Well, I've instead decided to forgo all the separate PC's/Workstations and figured why not just upgrade my server with a new CPU, few GPUs to handle all the additional VM's with passed through hardware and be off. So I give you my new all-in-one home storage/media server/virtual PC server.

BUILD PICS

Operating System

Unraid 6.8.1 (Nvidia Build)

Components

CPU: AMD EPYC 7452 32c/64t CPU
MoBo: ASRock Rack EPYCD8
RAM: 128GB DDR4-3200 RDIMM (16GB x 8)
GPU: GTX 1660 (for hw transcoding)
GPU: GTX 1060 (for main daily PC VM, Emulation Gamestreaming)
GPU: Quadro K4000 (for wife's daily PC VM)
HBA: Quad External SAS card to connect 2 DAS boxes

Storage

NVMe: Intel Optane 900p 280GB (x2) - For docker appdata & VMs
NVMe: Samsung 960 Pro 1TB - Write Cache
NVMe: Samsung 960 Pro 512GB - passed through to my daily driver PC VM
SSD: Micron 5100 Pro 1TB (various fast access folders, ie; Plex Sync, Emby Sync, nextcloud, etc.)
HDDs: 8 x 10TB WD Gold drives, 8 x 10TB WD White Label drives (shucked)
USB HDD: 10TB WD White label shucked and put in cooled enclosure (for torrent seeding)


So this server acts as the main computing hub for our entire house. We have two Microsoft Surface Pro 3's in the house. One is docked to a monitor/keyboard/mouse for my wife to stream her VM desktop via Moonlight. The other we just use for web surfing and what not on the couch. I have video and USB cables running from the server directly to my desk in my office for my main PC/VM. Currently I also use my main PC/VM to stream Launch Box / Big Box and other PC games to both my Nvidia Shield TV's in the house. I would like to have these be separate VM's but for the time being I can't fit any more GPU's in this system (physically) until I can replace the two dual slot cards I'm using with suitable single slot cards both for the gamestreaming and for HW Transcoding in Plex/Emby.


Power Numbers

Idle power usage with all array drives (16 of them) spun down is 165w. With all 16 drives spun up idle power usage is 225w.

100% load with cTDP set to 180w the system pulls 350w.


Benchmarks

I ran a few quick Passmark CPU Mark benchmarks today just get an idea of how the system stacks up.

This was a single baremetal instance.

Next I ran two simultaneous 32 thread instances in separate Windows 10 VMs. Though I realized after the fact the second VM only had half the RAM as the first VM. Not sure if that affected the score at all.

And then lastly I ran 4 simultaneous instances in 4 Windows 10 VMs each with 16 threads a piece. I didn't take screens of those but the combined total was 72,670.

 

[am45931472]

Noticed that you've gone for AMD EPYC Rome, How do you like its performance for those Win 10 Vms. Are thoes win 10 VMs "fast", snappy, quick, low latency. I have debated EPYC Rome or Threadripper 3000 for Windows 10 VMs. The clock speed is just so much higher on threadripper or consumer hardware, but the lack of RDIMM for a server just kills.

 

[IamSpartacus]

Noticed that you've gone for AMD EPYC Rome, How do you like its performance for those Win 10 Vms. Are thoes win 10 VMs "fast", snappy, quick, low latency. I have debated EPYC Rome or Threadripper 3000 for Windows 10 VMs. The clock speed is just so much higher on threadripper or consumer hardware, but the lack of RDIMM for a server just kills.

Absolutely no issues with the VM's. They are very quick and snappy. I'm not a huge PC gamer (not gaming at like 4K 120Hz or anything but the few games I've tried have had no issues at all. I'm seeing Max boost clocks 3.30-3.35GHz on up to 16-24 threads being fully loaded so I'll just about always see that speed on these VMs.

I went EPYC Rome over Threadripper for a few reasons (not in order of importance).

1. Power usage. Power costs are very high in my area (over $0.25 per kwH).
2. Related to the power issue is heat/noise as my server sits in a closet in my office/guest bedroom.
3. PCIe lanes. I have 3 GPU's in this system and will be eventually adding a 4th. I also run 4 x NVMe's, and an x8 HBA. Right now that's x72 in PCIe lanes before adding the 4th GPU. And while yes it's unlikely I'd be PCIe lane restricted on the Threadripper beacuse I'd have to max out all those devices at the same time (highly unlikely), I'm not a fan of being over provisioned in that department. It would always make me wonder when troubleshooting issues.
4. Motherboard options. It's pretty much impossible (I haven't seen one yet) to get a Threadripper 3000 board with IPMI on it. That's an absolute deal breaker for me. I use IPMI so much with how much I like to tinker. Can't live without it.

 

[MRK]

For me a impressive build. May I ask you something. How do you passtrough the GPUs? Tought its not easy as long you are not using Quadros. But it looks like you are using your normal GTX for the gaming VM.

 

[IamSpartacus]

For me a impressive build. May I ask you something. How do you passtrough the GPUs? Tought its not easy as long you are not using Quadros. But it looks like you are using your normal GTX for the gaming VM.

Quite easy using KVM. Pretty much just need to select it in the VM options. Its harder in VMware but can still be done.

 

[ReturnedSword]

@IamSpartacus You are a crazy man for sure. Everything about this setup is perfect and well thought out.

I always tell my SO that this hobby is what keeps me out of trouble elsewhere

 

[IamSpartacus]

@IamSpartacus You are a crazy man for sure. Everything about this setup is perfect and well thought out.

I always tell my SO that this hobby is what keeps me out of trouble elsewhere

Thanks for the kind words . And yes, my wife definitely likes that this hobby keeps me home instead of out and about .

 

[Navy_BOFH]

Well - here I was looking at a 10GBe NAS and then a small powerful VM host server... and your thread came along. I know what I am doing next!
What case did you stuff everything into? Something that shows HDD status and hot-swappable, or something desktop oriented with a ton of space?

 

[IamSpartacus]

Well - here I was looking at a 10GBe NAS and then a small powerful VM host server... and your thread came along. I know what I am doing next!
What case did you stuff everything into? Something that shows HDD status and hot-swappable, or something desktop oriented with a ton of space?

Check the image gallery link in my OP. I unfortunately don't have space for a full rack in my 3 br condo right now so this is all sitting in a closet on a metal wire shelf. Thus I can only fit a max of 18" depth chassis. So you'll see I use a compute box (4U Norco) for the CPU/MoBo/RAM/GPUs/NVMe's and then I connect to 2 DAS boxes running in U-NAS NSC-810a chassis'. I rather like having the compute and bulk storage separate. Even when I move into a house in the future and have room for full depth servers I will probably keep them in separate boxes likes this.

 

[IamSpartacus]

@epicurean Here you go. Don't want to hijack that thread. There is some custom stuff in there like mapping the Plex Sync directory to a different disk (to keep it out of appdata backups).

 

[fossxplorer]

How Ddo you use the desktop VMs?

 

[IamSpartacus]

How Ddo you use the desktop VMs?

Mine I use with cables directly plugged into the GPU and USB ports. I also stream my VM to my Nvidia Shield TVs to use Launchbox Big Box emulator on my TVs. My wife streams her VM to a docked Surface Pro 3 using Moonlight.

 

[pututu]

I'm seeing Max boost clocks 3.30-3.35GHz on up to 16-24 threads being fully loaded

Congrats on the nice build! Do you by any chance know the CPU clocks when all the 32 threads are running with non-AVX workloads? How about the AVX workloads (e.g. Prime95)? Thanks.

 

[IamSpartacus]

Congrats on the nice build! Do you by any chance know the CPU clocks when all the 32 threads are running with non-AVX workloads? How about the AVX workloads (e.g. Prime95)? Thanks.

I forgot which test this was, sorry.

 

[fossxplorer]

Wow, 32C@155W it is simply masterpiece by AMD!

 

[pututu]

All cores at 3.2+GHz is awesome for this chip. Thanks for sharing!

 

[sno.cn]

You're getting much better load frequency than the numbers I posted. That's fantastic.

 

[Trotador]

You're getting much better load frequency than the numbers I posted. That's fantastic.

So @sno.cn you are nott getting the all core turbo clock of 3250MHz in your H11DSi? I'm tempted to go to an equal set-up but i'd like the maximum boost frequency for number crunching ,

 

[sno.cn]

I get 2850 MHz give or take, running 128 threads and heavy cache saturation, and 180w cTDP. I don't know how much cross-socket link contributes to power envelope, but I have another 7452 build on H11SSL-i that I'll test when RAM gets here tomorrow.

 

[John Burns]

@IamSpartacus How well does moonlight work for game streaming? The website claims 4k @120 fps, but my past experience with streaming makes me skeptical. My server closet is just far enough away that using dedicated cables won't work without using a extender that compresses.