All in one home server build for Proxmox

[alvitali]

Hey everyone, hope you had a good weekend

Newbie at PC / server building here. I’m looking to build my first proper home server. At the moment I’m running a Synology NAS + several (Mini) PCs running containers + OPNsense.

I really want to get rid of the Synology, and take this chance to rebuild my entire setup with virtualization in Proxmox. My plan is to build a single, powerful machine which can handle running TrueNAS with 8-12 disks, 4-6 VMs with about 80 containers between them + potentially virtualized OPNsense in the future.

I’d like to prioritize power efficiency, as power is quite expensive here (current setup is using 250W so anything less would be great). This had me look at consumer gear, however I’d really want ECC (as that’s what I had on the Synology), as well as some out-of-band management (IPMI or Intel AMT etc.)

My first pick was to look at an Intel platform with integrated graphics due to the energy efficiency of Quicksync for Jellyfin (most used application atm). Due to my wish for ECC, this limits motherboard availability considerably. So, my preferred setup atm is:

OS: Proxmox
CPU: Intel Core 12th / 13th Gen (not sure yet, will have a look on 2nd hand offerings with iGPU)

• Motherboard: Something with the Intel W680 chipset:
  Supermicro's board looks amazing: X13SAE-F | Motherboards | Products | Supermicro
• Asrock has one that’s basically unobtainable: https://www.asrockrack.com/general/productdetail.asp?Model=W680 WS#Specifications
• And Asus has one that looks great, but doesn’t really support IPMI (separate HBA) and not so great reviews: Pro WS W680-ACE IPMI｜Motherboards｜ASUS Switzerland

Chassis: I bought a 2U server chassis some time ago with SAS / SATA backplanes with SFF-8087 connectors.
Drives: Seagate Exos X18 currently running in my Synology
RAM: I've got 2x32GB Crucial UDIMM I bought in summer (thank god, else I could scrap this build) - I wanted to get another 2x32, but that will have to wait now ig... kind of bummed out as I won't be able to get the same sticks anymore with Crucial shutting down
Add-in Cards:

• Graphics card for running LLMs, probably 2nd hand
• Intel X520-DA2 10GbE NIC currently used in my Synology
• SATA / SAS HBA for my chassis backplanes

Supermicro's MB would be my first choice here as it seems perfect for what I want. However, price is quite high, and availability is a bit uncertain where I live…
This made me think of considering AMD, even though integrated graphics don’t seem to be as great.

This motherboard e.g. costs 2/3 of the Supermicro one and seems okay for my needs, paired with an AMD Ryzen 9 7900: https://www.asrockrack.com/general/productdetail.asp?Model=B650D4U#Specifications

After adding a graphics card, 10G networking, would already fill up all of my PCIe slots if I also need an HBA for the SATA / SAS connectors.

Is this a good direction or am I going at this the completely wrong way? I’ve thought about using old enterprise gear, however I’m completely lost there – I have no idea which gear is somewhat energy efficient, and it seems impossible to understand how the Intel Xeon lineup works, which line, generation etc.

Really happy about any feedback!

 

[louie1961]

What kind of UDIMMS did you buy? DDR4 or DDR5? If I was building something on a budget right now I would probably go DDR4 and AM4 to reduce the cost. Something like this:

AsRock Rack X570D4U-2L2T OR

AsRock Rack X570D4U

AMD RYZEN 5 PRO 5650G OR

AMD Ryzen 7 5700X (trying to stay 65 watts in the CPU...that would be my preference)

NEMIX RAM 128GB (4X32GB) DDR4 3200MHZ

If you don't want to go with the Ryzen5 Pro CPU, then fo Ryzen 7 and add in a video card. The problem with an mATX boardis a lack of PCI slots. The first board has built in 10gbe networking (but copper, not fiber), but that leaves you with a PCI x16 slot for a video card, and an X8 slot for your HBA. If you go with the other board, you need to have a CPU with built in graphics that also supports ECC memory. That means a Ryzen 5 or 7 Pro. Unless 8 sata ports is enough to get you by, then yoiu can forego the HBA


If those are DDR5 UDIMMS, they could fund half this build if you re-sell them. I personally wouldalso skip the backplane. How often do you really need to swap disks? That is just going to take up space and use a bit of power, for something I personally wouldn't find very useful.

Right now I am running 7 VMs (one of which is TrueNAS Scale) and 13 containers on a Ryzen 5 Pro 5650GE (35 watt part) with 64gb of RAM. My CPU is just idling along at like 2% I have 8 SATA SSD drives and 3 NVME drives in this machine, plus a X520 NIC and all in it draws about 40 watts idle. I have TrueNAS, a docker host, three wordpress instances (each in their own VM), Home Assistant OS and a discourse forum all running as VMs. I don't do anything with a media server, so I am not concerned about transcoding. My websites don't get a tremendous amount of traffic, but some. I also run pfSense as a VM but on a different machine. I would strongly encourage you not to put your router/firewall software on your main/only server. It's a bit of a nightmare to troubleshoot when something breaks.

 

[alvitali]

Thank you for sharing your build, I was afraid 6 / 8 cores was not going to cut it, but seems like plenty for my use case! Do you use ZFS in TrueNAS?

If those are DDR5 UDIMMS, they could fund half this build if you re-sell them.

Yeah, I got DDR5 @4800MHz... Looking at the prices for (new) DDR4 UDIMMs, this doesn't really seem compelling either atm :/ Sadly DDR4 is not that much cheaper here in Europe, and it's hard finding used 16GB+ sticks locally anyways.

Is there not that much of a performance difference between DDR4 and DDR5 when doing virtualization & self-hosting / serving content?

The AsRock Rack X570D4U options seem great especially with those 8 SATA ports, which is all I need at the moment. I wish I had started building this sooner ^^

trying to stay 65 watts in the CPU...that would be my preference

Is the max TDP that significant for idle power usage? I assumed running idle wouldn't make that much of a difference between that and a CPU with higher max draw.

Man, I hate Intel and all the stuff they do, but looking at video transcoding benchmarks they just blow AMD out of the water... literally the only thing they have an edge, which still makes me prefer an Intel build. But at this point I might just get an ARC A310 for video transcoding, which shouldn't draw a lot of power at idle? That would necessitate another build if I want to run LLMs in a different VM in the future though...

 

[louie1961]

Thank you for sharing your build, I was afraid 6 / 8 cores was not going to cut it, but seems like plenty for my use case! Do you use ZFS in TrueNAS?

Yes I do. But I only have 4 drives in TrueNAS. The other 4 are used as the rpool in Proxmox at the moment. My motherboard is an ITX board GIGABYTE B550I AORUS PRO AX It has 4 onboard SATA ports that I use for Proxmxox. I use an inexpensive M.2 NVMe to SATA adapter (which has 6 ports) passed through to TrueNAS. Works great.

Is there not that much of a performance difference between DDR4 and DDR5 when doing virtualization & self-hosting / serving content?

I don't know the exact delta to be honest, but my system is plenty fast for what I do. My understanding is for typical home lab use, most VM and container workloads are not memory bandwidth-bound. They're typically constrained by storage I/O, network, or CPU. Claude.AI suggests the performance difference is 1-2% at best.

Is the max TDP that significant for idle power usage? I assumed running idle wouldn't make that much of a difference between that and a CPU with higher max draw.

No real difference in idle power, but less power draw under load. The AMD Ryzen 7 5700X will give you 95% of the performance something like a AMD Ryzen 7 5800XT while using much less power under load. The 5800XT only benefits you if you are regularly pegging all of the cores. Otherwise, for VMs and containers,the extra horsepower isn't needed in a home lab.

I would get a cheap Nvidia card. If you have no plans for LLM workloads, a cheap 1000 or 2000 series Nvidia is very easy to pass through to docker containers and is more than enough for transcoding as well as simpler AI tasks like facial recognition in Immich or Photo prism.

That would necessitate another build if I want to run LLMs in a different VM in the future though

No, I don't think so. I think it is as easy as swapping out to a more powerful GPU. LLMs require lots of GPU memory to really run well. But the CPU and memory combination I laid out above isn't going to hobble you, IF you get a decent GPU. You may want to just jump to something like Nvidia RTX 3090, RTX 4090, or even a RTX 5060 (cheaper alternative with less memory) all will idle in the 15-25 watt range. The max power draw will only be an issue under load.

 

[gea]

The first decision for a Proxmox All in One is whether you want a storage VM or not. Proxmox is basically a Debian optimized for Hypervisor use with newest ZFS integrated and a web-gui for VM, system and basic ZFS management. A Debian Storage VM like TN means full OS virtualisation of a Debian with ZFS and SAMBA for SMB ontop the same Debian with the same ZFS that can run the same SAMBA or the faster inkernel ksmbd SMB server.

Option 1, Proxmox as barebone NAS:
Just install samba (or ksmbd) and acl as you would do on any other Linux. If you need a web-gui for advanved ZFS, share and ACL managementl look at web-gui add ons like Cockpit/Poolsman or the copy and run multi-os/multi server napp-it cs.

Option 2, Proxmox + Storage VM (Free-BSD, OmniOS, Linux, Windows)
This gives additional tools and management options. To access the storage from Proxmox, you need network shares like NFS or SMB what reduces perfomance (sequentially and latency) of VM virtual disks compared to direct access to zvols on a local Proxmox ZFS pool. You need extra CPU resources and RAM for full virtualisation and you must maintrain two different distributions regarding bug and security fixes.

Option 2a: VM Storagepool on Proxmox and a second ZFS pool on the storage VM
This gives full performance for VMs and the options of the storage VM for other filer and backup use
Main disadvantage is that ZFS wants RAM then for performance or special vdevs twice on both systems.

Option 2b: All Storage on the storage VM, only a Proxmox rpool for OS
This makes ZFS management easier. You should provide the resources and must accept a lower VM performance.

 

[alvitali]

Okay, thank you for your help! I spent a few more hours looking at all of the options and weighing them up.

Sadly DDR4 is not that much cheaper here in Europe, and it's hard finding used 16GB+ sticks locally anyways.

So, while DDR4@3200 costs almost as much as DDR5, I found DDR4@2400 MHz sticks with 16GB for "only" around 60$ each (ECC UDIMMs). This would make an AM4 build viable, but I'd still spend a few hundred bucks on just motherboard + RAM for a "legacy" platform. If RAM was cheaper that would be a non-issue, but only getting 64GB of RAM (same as I have right now on my baremetal machine) while still spending 200+ $ for a used motherboard rubs me the wrong way.

So, my current thinking is:
I can get a Asrock B650D4U for around 250$ used, which I'll pair with an AMD Ryzen 7 8700G (TDP of 65W, and seems to have a great iGPU, including AV1 encoding), which I can get for 200$ used right now.

Only gripe with this choice is having just 4 SATA ports on the motherboard, which seems to generally be the case for the AM5 platform... AM4 has tons of boards with 8 SATA ports meanwhile :/ which means I'll have to stick a SATA PCIe expansion card in. However, if I want to use the 16x slot for a graphics card I'll only have a 4x slot left, which has me looking at a card like this... all of the HBAs with SFF connectors seem to need an 8x slot.

The other option would be to go with aforementioned Supermicro + Intel Core chip of 12/13/14th gen, but that would cost 200$ more, although I'd get all the PCIe expansion I'd want?

I think it is as easy as swapping out to a more powerful GPU

I assumed if I want to passthrough a GPU to my VM running ollama, while also using a card to passthrough to Jellyfin running in another VM... but I guess I'll cross that bridge when I get there ^^

 

[alvitali]

@gea thank you for your writeup on storage options!

I envisioned doing Option 2a – I assumed I'll want to run ZFS on the storage volumes (set up in a TrueNAS VM) for data integrity purposes. But I meant running databases on ZFS is not a good approach due to all the read / writes involved. Or would reducing the block size make this a non-issue?

My plan was using spinning disks for large file storage in TrueNAS (file storage, backups media) and then creating LVMs for container / VM storage? (i'll have to read up on proxmox volumes again)

 

[gea]

LVM on ext4 has one advantage, this is a slightly lower write amplification compared to Copy on Write ZFS.
This can be a plus with cheap desktop ssd (what you should avoid anyway)

In all other aspects (performance due a superiour ZFS read/write caches, VM security with sync, checksums for bitrot protection, crash resilency with Copy on Write, Snaps, hybrid ssd/hd pools with special vdev, compress and encryption, .., ZFS filesystems and ZFS zvols as virtual disks are superiour.

Databases and VMs on ZFS is fine, use flash storage with a smaller recsize like 32-64K.

 

[louie1961]

If you are going AM5, I personally would go with the supermicro board for AM5 Supermicro H13SAE-MF Motherboard Micro-ATX AMD EPYC™ 4005/4004 series, Ryzen™ 9000 & 8700G* & 7000 Series single socket LGA-1718. - Newegg.com

And I would pair it with an Epyc 4005 cpu like this one, instead of a Ryzen AMD EPYC 4005 (5th Gen) 4345P Octa-core (8 Core) 32MB L3 Cache Socket AM5 65W Processor - OEM - Tray Pack - 100-100001556 - Newegg.com

 

[alvitali]

If you are going AM5, I personally would go with the supermicro board for AM5 Supermicro H13SAE-MF Motherboard Micro-ATX AMD EPYC™ 4005/4004 series, Ryzen™ 9000 & 8700G* & 7000 Series single socket LGA-1718. - Newegg.com

That is a great board, how did I not find this yet

And I would pair it with an Epyc 4005 cpu like this one, instead of a Ryzen AMD EPYC 4005 (5th Gen) 4345P Octa-core (8 Core) 32MB L3 Cache Socket AM5 65W Processor - OEM - Tray Pack - 100-100001556 - Newegg.com

What is the benefit of the Epyc over the Ryzen – higher memory bandwidth? Or is it just that the motherboard won't support the iGPU anyways (didn't find any clear information in the manual regarding that)

This can be a plus with cheap desktop ssd (what you should avoid anyway)

What SSD lineup do you recommend? I remember looking at Micron / Samsung entry-level enterprise SSDs, but those were still really expensive even used

 

[gea]

The AI boost makes RAM and disks currently very expensive with prices will go up even more.
If possible prefer SSDs with powerloss protection, otherwise check dwpd or max write values.

If you cannot get one at a good price, use what is on offer. It may be not as fast or must be replaced earlier.

NVMe are faster than ssd but often cheaper

 

[louie1961]

What is the benefit of the Epyc over the Ryzen

The Ryzen part is optimized for gaming. Higher clock speeds, and less onboard cache. The Epyc part is supposed to be optimized for server workloads. Slightly lower clock speeds, but double the cache. Also more PCI lanes (24 vs 28) which may not matter on this board. The Epyc part also reportedly has more advanced RAS features (memory patrol scrubbing PCIe Advanced Error Reporting)