Build advice - virtualization server

[doppler_shift]

I'm trying to decide on a cost-effective virtualization server build. The major workloads are C++ compilation related; CI/CD, tests, GH actions runners, etc. - all big hogs in terms of CPU power which benefit greatly from single thread performance, especially since most of the C++ code is metaprogramming heavy (low TU count, long time to compile and link each TU). Then there are a couple low traffic PostgreSQL databases and internal admin/monitoring websites running on nginx/nodejs. In terms of memory and connectivity the requirements are pretty modest; 64-128GB of RAM, one high durability NVMe drive for all the VMs, one RI boot SSD, 10Gb NIC, and a few SATA ports for a ZFS HDD array for backups and bulk storage. I'm currently looking at two options;

• A Ryzen 7950X build with something like an ASRock B650D4U which on paper is very appealing because of the ridiculous single-thread performance, but in reality I've seen people having major issues with this platform in the past few months like random crashes, memory not working / systems not POSTing at all, etc.
• An EPYC Milan build with something like a 7313P and Supermicro H12 board which in comparison is rock solid but also has much lower single-thread perf and feels like overkill because I don't really need all the I/O and RAM capabilities of this platform

In terms of cost these two solutions are pretty close. I also couldn't find an Intel alternative at the same price point that gives similar performance in terms of core count vs. single thread perf.

 

[CyklonDX]

In terms of issues on new platforms issues are always given on them. Thought the issues you listed are likely due to crappy psu, overheating, too high clocks on sticks, and overheating nvme's.

While you can expect issues with kvm on new platforms for next few months, but you can solve listed issues by building your box with solid components.
*in short QVL memory sticks that are ECC, a decent seasonic psu, ups, decent heatsink on nvme, good set of fans, and cpu heatsink - then setting fan rpm in relalations to temps in following curve.

 

[doppler_shift]

After scouring forums and reddit, my general plan for the build:

• CPU: AMD Ryzen 7950X
• Motherboard: ASRock Rack B650D4U
• Memory: Micron 2x32GB DDR5-4800 ECC UDIMMs (I'd go for 2x48GB ECC UDIMMs but I can't find them anywhere, and also considered going 4x32GB but looks like currently going 2DPC is a waste of time on AM5)
• Chassis: Rosewill RSV-L4500U (4U)
• Cooler: Deepcool LT720 (mounted on the middle fan wall)
• PSU: Corsair RM750x (it's hard to source Seasonic where I live and this PSU has very good reviews/scores and also has two EPS12V connectors)
• High endurance NVMe drives for the VMs (Ultrastar SN200 / PM1725b / something else with 3 DWPD) connected via a 4x/4x/4x/4x U.2 bifurcation adapter on the x16 slot
• Pass-through the onboard SATA controller + an x1 PCIe 4.0 SATA adapter with an ASM1166 to a TrueNAS SCALE VM for bulk storage for a total of 4+6 SATA ports (or maybe a SAS3008-based card?)
• 25GbE NIC on the x4 slot which should be enough bandwidth for full-duplex single port 25GbE; full duplex is about 6GB/s and PCIe4.0 x4 is 8GB/s raw
• Boot drive for ESXi 8 on the M.2 slot that's connected to the chipset

 

[donedeal19]

I would love to see more of your build inside this chassis. How many drivers are you planning to use in your build, and what will your idle power consumption be like?

I also have a setup that is being used for similar functionality, with hard drives stuffed in it, and I'm thinking about upgrading a few parts.

 

[doppler_shift]

I would love to see more of your build inside this chassis. How many drivers are you planning to use in your build, and what will your idle power consumption be like?

I also have a setup that is being used for similar functionality, with hard drives stuffed in it, and I'm thinking about upgrading a few parts.

For the bulk storage probably going to start with 4 or 6 spinners configured as two-way mirror vdevs for a ZFS pool. For the VM storage I'm thinking one 3 DWPD drive for the ESXi store and then pass-through two fast/cheap 1 DWPD drives (think PM9A3) to the main PostgreSQL VM and configure them as a mirror.

Idle power draw is going to be pretty bad most likely; 7950X is a dual CCD processor and that means bad idle power consumption to start with, and I'm not sure how power optimized is the B6504DU board, but we'll see (I have a PDU with a power meter in my rack so I can report back).