Socket 3647 - still the best for home lab in 2023?

[Ivan Dimitrov]

I have been running 3647 platform pretty much since they were released and recently I got the upgrade itch. But I am pretty hard pressed to find a "stick to scratch it".
My current system is:

• Xeon Gold 6230
• 192GB RAM
• 3x Intel NVME SSD - 6 + 8 TB + 400GB boot
• 4HDDs
• nVidia 1660 Supper (considering adding second GPU for another gaming VM)

With normal load it consumes ~140W. Of course more when gaming or the system is loaded.
The clarify my requirements/priorities are:

• low power consumption - living EU explains a lot + the server is next to me --> need to be low noise.
• low IDLE power consumption - the server is not running anything taxing all the time, so idle/low load power consumption is essential
• As much PCIe as possible - I am using it to host VMs for network play with pcie passthrough + old enterprise HW usually uses wider but slower PCIE interfaces
• A lot of RAM - the above mentioned gaming VM/VMs + zfs + playing with k8s cluster need it
• Reasonable price (it is a homelab after all)

After exploring the alternatives I am a bit disheartened - none of the current platforms seem to provide reasonable alternative or significant improvement compared to what I already have. The short analysis is:

• Current Intel/AMD server platforms, W790 included - simply too expensive, power hungry and I don't really need the new features (PCIE 5, DDR5, 50 cores, etc.)
• Rome/Milan EPYC
  • high power consumption
  • PRO: able to reuse my current RDIMMs
  • PRO: more than enough PCIE lanes
• socket AM5
  • limited choice of "server" grade motherboards with multiple PCIe slots
  • idle power consumption seems to be higher that than the Intel alternative
  • 192GB RAM support is questionable at least with the current CPUs
  • DDR5 is still at premium
• socket 1700
  • a few boards with multiple PCIE lanes (Pro WS W680-ACE, H13SAE-MF). Can make such configuration work
  • DDR5 is still at premium
  • CON: power hungry at high loads

Basically any of those options will be ~1000/1500 EUR and at the end of the day I will get system which does pretty much the same. For some of the platforms I am not even sure I won't get increased constant load power consumption.

Am I missing something or really the 3647 is still the homelab king? For the money I will for sure get better performance but I am not sure I will get better performance per W. Compute performance is not limiting factor for me today and with the new system I will get reduce my flexibility.
How do you see, have considered similar upgrade?

 

[nexox]

I recently chose to to with a dual 3647 system for my new workstation build for more or less the same reasons you're considering, everything better was quite expensive (and I chose to start with cheap 6132s while I wait for the Cascade Lake refresh chips to come down in price a few years from now.)

I don't have the thing entirely built yet, no GPU or 10G NIC yet, but idle power is only 100W, perhaps you could tweak some settings or upgrade some components to allow more portions of your CPU to reach lower power states.

 

[i386]

• As much PCIe as possible
• A lot of RAM

(for me) this eliminates everything but amd epyc systems

high power consumption

I went from a single xeon e5 v4 4core and 2 dimms system to a single epyc zen 3 24cores with 8 dimms and the difference was less than 50 watt for the same workload* (about 200 watt from a 24 bay sm chassis with 17 hdds and 2 sas ssds + raid controller + mellanox 100gbe nic)
if we say that a dimm consumes 4watt that's 24 watt (6*4) from the additional dimms and about 26 watt from the amd epyc platform.
I think 26watt for 20 additional cores, pcie 4.0. and a lot more memory support is not much

*measured with an apc ups

 

[jei]

Atm cannot see only the server, whole rack:

H12SSL
- EPYC 7443 24-Core
- 8x 3200 RDIMM
- 8x U.2 NVME
- 2x 7k2 HDD
- 3x Add-on, Oculink, 10Gbit
- 10+ fans

ICX7150-C12P

2x R710 PoE

->

200W on low load/idle.

240W+ including Eaton 9SX Online UPS (pf 0.8-0.85)

There's different kinds of EU. Last months was 5-10c/kWh.

 

[Jorge Perez]

Why not go for Xeon D based servers?

Power consumption is similar to desktop CPUs.

Looks like the 14 core is $4000

 

[aosudh]

Low power consumption + tons of requirements, the only platform i can think of is ARM , especially ampere altra. However, their price isn't fair enough

 

[Ivan Dimitrov]

@i386 - Thanks making me look at the older EPYCs (Rome/Milan) one more time. They really offer much more room for growth as a platform and performance. There are also some really cheap options for CPU+mobo on eBay. Based on STH reviews a 16 core Rome will be roughly on the same performance level as I have currently not to mention the potential for more cores and further upgrades to Milan. But as power consumption I doubt I will be able to fall much lower. It appears there is even threat on the forum here It is for Milan specifically but but Rome sin't that much different.

@Jorge Perez - Yeah the Xeon Ds are looking tempting but they offer very little PCIe lanes and the price premium is too much

@aosudh - The ARM doesn't seem to be that mature still + software compatibility... Also they are power efficient considering compute/W. But at least in some of the reviews I have watched/read their power management seems to be lacking (see platform maturity).

 

[Jorge Perez]

@Ivan Dimitrov Why not get multiple Xeon D systems, that would give you more PCI-E

 

[Stephan]

3647 is cheap-ish, extensible, performs well, stable, so really one of the kings. Difficult to get better without major investment or incur less features. Maybe you can send the HDDs into standby to save on power and noise. Like 15 watts.

Next upgrade shall be a machine with single thread performance alot closer to apple m2. Or whatever datacenter RISC-V monster the chinese crank out, who are fed up with ARM and x86 US handcuffing. Their MIPS arch is getting long in the tooth. And now that SMIC can do 7nm (ok they didn't say how good the yield was). With Coreboot, PCIe 4+ and more threads you will ever need. Possibly with AI acceleration so you can train GPT-5 at home within a week or two if you buy the RAM.

I had some hope for ARM but RPi is just too limited and others can't or won't do Skylake performance with ECC. Time is running out for ARM.

 

[bayleyw]

Skylake-SP is very good, as @Stephan said you get nice stable performance on a modern-enough platform (critically, everything Skylake and newer supports NVMe boot and bifurcation which was hit and miss on its predecessor, and NVMe and bifurcation are really useful on home systems). You might also consider Ice Lake just for PCI-e 4.0 support which opens up 2:1 consolidation of drives and good 100G networking (PCI-e 3.0 100G NICs are these crazy x16 single port contraptions). ICL also supports a few more DIMM slots and some extra AVX features for good measure.

Homelabbing is what the enterprise guys would call 'edge' computing - density is less important, the machine might be idle sometimes, and you might have to interface with random peripherals which aren't on the latest I/O standards. Alas the latest offerings are all density-focused datacenter products, and the optimizations for datacenters at scale are orthogonal to the ones for high I/O density and low idle power.

I think even CPU vendors are realizing this, Epyc 'Siena' has been known for some time and is a 64xZen 4c variant of Genoa designed for low-density deployments. Unfortunately it is still chiplet based which means it will have a really hard time scaling down, but Zen 4c chiplets are very dense (16 cores per die), so maybe there will be a 1+1 16c variant that should cut down on in-package fabric power...

 

[drdepasquale]

The price to performance ratio of LGA 3647 platforms is still high compared to EPYC

 

[Ivan Dimitrov]

Yeah, it seems I will be dragging s.3647 for now, until something better appears. And would you know it AMD might have just announced the answer - Siena

 

[pyite]

My $.02 here... I have been using LGA 3647 for awhile in a couple of systems at home. The 6154 CPUs were being e-wasted at work so they were $0, which helps.

This is offset by how expensive the motherboards are. Also, this socket can only do PCI-e Gen 3, but that isn't much of a problem usually.

Probably the most affordable option is a bare bones HP Z6 G4. Be careful - lots of proprietary pieces so get one that has a memory cooler and the front piece with the power switch, SD slot, etc. The separate components are very expensive on eBay. Otherwise it works quite well.

I also have a SuperMicro X11SPL-F. The post times are long, which is generally a big fail on server boards. Apart from that it is excellent. I just put this board in my old fanless Zalman TNN-500AF and it great so far. It will eventually overheat if I punish all the cores but it can at least build Linux kernels and play 3D games with no issues and total silence.

 

[NPS]

It also seems to be difficult to find PCI-e gen 4 support. Our 3647 servers at work do Gen 4, but any Gen 4 capable motherboard seems to be big $$$.

There is no gen4 for LGA3647 at all so I don't get what expensive options you are talking about.