AMD EPYC Questions

[JiKeParty]

For those running Proxmox with Linux VMs and a few lightweight containers 24/7, what has been your long-term experience with AMD EPYC 7002/7003 systems built from retired datacenter hardware?


Specifically:

• Which motherboard did you choose?
• How reliable has IPMI been?
• What idle power draw are you seeing at the wall?
• Any issues with PCIe passthrough for GPUs, NVMe drives, or HBA cards?
• Are there any compatibility problems with Proxmox updates, ZFS, or Linux kernels?
• If you were building again today with a budget around $1,000-$1,500, would you still choose EPYC over newer Ryzen systems?

 

[Auggie]

I have been through this exactly (7402 from @Tugm4470 here on the forum)

H12ssl-i
extremely
100w+, but that's fully loaded ram and a couple u.2 ssds
only with amd graphics cards
no
no, single thread is fairly slow these days

was originally on proxmox with zfs using machine as a higher end nas, was overkill for that use case
machine is now an llm inference server, works pretty well for that, especially given price

 

[JiKeParty]

I have been through this exactly (7402 from @Tugm4470 here on the forum)

H12ssl-i
extremely
100w+, but that's fully loaded ram and a couple u.2 ssds
only with amd graphics cards
no
no, single thread is fairly slow these days

was originally on proxmox with zfs using machine as a higher end nas, was overkill for that use case
machine is now an llm inference server, works pretty well for that, especially given price

Wow! I'm so surprised to get a great answer. Thank you!

 

[Electrified]

Depending on your electricity cost a Ryzen / Intel Core system might make more sense to cover all your 24/7 Stuff,
i usually recommend people to get two systems, one low power one that handles all the 24/7 stuff and then a second high power (Epyc / Scalable / E5 v4) one for demanding stuff / test environments that is only powered on when needed.

As for your questions,
my favorite Epyc Board is definitely the Gigabyte MZ32, i am using several of them for years and they have great expandability / lots of pcie / 2dpc and are single socket so idle power isnt too bad (about 90w for a minimal system)
Other decent ones i have used so far, Gigabyte MZ31 (only PCIE gen3 but can be had for about 250€ on ebay right now), Asrock Rome d8-2t (good if you can only fit an ATX and not EATX in your case) and Supermicro H12ssl series is also decent

IPMI is great and basically the same on all of these

Full idle power draw in a normal setup at least 100w, but if you got some stuff going on like even just a few VMs running it can prevent lower power states and i would probably plan for around 150w, if you have a bunch of pcie or storage connected then more.

Proxmox and ZFS support is great (i mean its literally designed for platforms like this)

For 1k i would go with 2 systems like i mentioned at the top, also it is hard to recommend a specific setup without knowing your resource usage.
"Linux VMs and a few light containers" can mean the basic selfhosting stuff that uses less than 32gb ram total and will run fine on basically any ryzen / office pc from the last couple years or load an entire epyc system if you are doing local ai, vdi, test / learning environments.

How many pcie slots do you think you will use?
Are you planing to add gpus?
What are you currently running and how much is your average cpu, ram and storage usage?

Also keep in mind you can get server grade mainboards with nice ipmi for consumer cpus like Ryzen (Gigabyte MC12, Gigabyte MC13, Supermicro X12-SCZ-F...) and you can get them down in the 20-30w range in idle

 

[JiKeParty]

Depending on your electricity cost a Ryzen / Intel Core system might make more sense to cover all your 24/7 Stuff,
i usually recommend people to get two systems, one low power one that handles all the 24/7 stuff and then a second high power (Epyc / Scalable / E5 v4) one for demanding stuff / test environments that is only powered on when needed.

As for your questions,
my favorite Epyc Board is definitely the Gigabyte MZ32, i am using several of them for years and they have great expandability / lots of pcie / 2dpc and are single socket so idle power isnt too bad (about 90w for a minimal system)
Other decent ones i have used so far, Gigabyte MZ31 (only PCIE gen3 but can be had for about 250€ on ebay right now), Asrock Rome d8-2t (good if you can only fit an ATX and not EATX in your case) and Supermicro H12ssl series is also decent

IPMI is great and basically the same on all of these

Full idle power draw in a normal setup at least 100w, but if you got some stuff going on like even just a few VMs running it can prevent lower power states and i would probably plan for around 150w, if you have a bunch of pcie or storage connected then more.

Proxmox and ZFS support is great (i mean its literally designed for platforms like this)

For 1k i would go with 2 systems like i mentioned at the top, also it is hard to recommend a specific setup without knowing your resource usage.
"Linux VMs and a few light containers" can mean the basic selfhosting stuff that uses less than 32gb ram total and will run fine on basically any ryzen / office pc from the last couple years or load an entire epyc system if you are doing local ai, vdi, test / learning environments.

How many pcie slots do you think you will use?
Are you planing to add gpus?
What are you currently running and how much is your average cpu, ram and storage usage?

Also keep in mind you can get server grade mainboards with nice ipmi for consumer cpus like Ryzen (Gigabyte MC12, Gigabyte MC13, Supermicro X12-SCZ-F...) and you can get them down in the 20-30w range in idle

Thank you for the thorough and thoughtful response!