[WTB] 2 x VM servers, flexible needs

[altano]

Here's a puzzle for you all. I want to upgrade my personal VM servers. I currently have an EPYC 3251 and a Xeon-D in a Proxmox cluster, and I'm getting impatient with build times in my personal software development (single core clock speed bottlenecked).

My requirements:

• 2-nodes (4 would be fun, if it doesn't go over my power budget)
• More CPU power (mainly single-core) than the EPYC 3251 (Single Thread Rating=1,872 / CPU Mark=13,965). I'll take a lot more power if I can get it, but I'd settle for ~25-50% faster builds.
• RAM = 64GB/node min, but 128GB would be lovely.
• Room for 1 GPU/node. I'm currently using a Quadro P2000 but I'd like to upgrade, so I can buy around the requirements of the server.
• Budgets:
  • 3U room in rack
  • ~750W total power (could maybe squeeze out a little more)
  • ~$4k total price. For really nice servers, e.g. EPYC, I'd be willing to go a little higher. I'd be very happy spending a lot less, though, e.g. on Xeon v4s that are dirt cheap, if I can balance getting a performance boost while not blowing my power budget

I'd be happy to hobble together parts from various places or buy wholesale. I'll take deals for trash on eBay. Intel/AMD, # of sockets, timeline: I'm flexible on all of it.

How would you all approach this? How can you get tons of bang for your buck if you're as flexible as I am?

 

[Samir]

So I wouldn't look at servers since you need single thread performance. Seems like one of the highest single thread budget modern cpus is the i3-12100, and it should feel roughly 2x as fast as the epyc even though it's only 4c8t:

AMD EPYC 3251 vs Intel Core i5-11600K @ 3.90GHz vs Intel Core i3-12100 [cpubenchmark.net] by PassMark Software

But if single thread is what you need, a pair of these in desktop rack cases would run pretty cheap and cool, not to mention cheap. And ram might be cheap too depending on what you can find.

 

[altano]

Interesting! I’m not exclusively doing software development: I occasionally transcode video, run game servers, run a number of VMs/containers doing various small tasks, etc. I’m not sure 4c/8t per node would still be sufficient for all of that, but it might be. I hit the single core clock bottleneck way more often than I ever use all 8 cores right now, for sure.

If you were going that route, and you were racking the gear, what sort of desktop rack cases would you use? (Keeping in mind my 3U total room available)

EDIT: Also, going the i3 route would mean no ECC memory and no ipmi-capable motherboards, right? I’m racking the gear in a datacenter so ipmi is non-negotiable. Requirement scope creep

 

[Samir]

Interesting! I’m not exclusively doing software development: I occasionally transcode video, run game servers, run a number of VMs/containers doing various small tasks, etc. I’m not sure 4c/8t per node would still be sufficient for all of that, but it might be. I hit the single core clock bottleneck way more often than I ever use all 8 cores right now, for sure.

If you were going that route, and you were racking the gear, what sort of desktop rack cases would you use? (Keeping in mind my 3U total room available)

EDIT: Also, going the i3 route would mean no ECC memory and no ipmi-capable motherboards, right? I’m racking the gear in a datacenter so ipmi is non-negotiable. Requirement scope creep

Hmmm...I somehow forgot that you wanted to upgrade the existing servers. Depending on which xeon d you have, you may be able to replace everything with a dual cpu rack server or workstation. That definitely might be a bit more in cost than some i3 setups, but I think it would fit the bill a lot nicer since you need the ipmi.

 

[ScottishTom]

Can you give us some idea of the workflow, like do you build exclusively on one of the servers and run tools etc on the other, or is it all spread evenly among the two boxes? Just wondering if it would be worth it for you to have a single "slow" box (which would be adequately filled by one of your existing machines) and then a faster "build" box.

Anyways, if you're struggling for single-core performance (know how that goes), I'd consider E-series Xeons or something like an ASRock Rack X470/X570 boards with a suitable Ryzen. Either option would support up to 128GB RAM if paired with an appropriate motherboard (IPMI + ECC totally an option with either), and these will generally clock higher than a server CPU at a given $ and power-budget.

 

[Branko]

This can be good start

AMD EPYC 7551P CPU 32 Cores + Supermicro H11SSL-i Motherboard +4x 32GB 2133P RAM | eBay

AMD EPYC™ 7551P. 1x Supermicro H11SSL-i motherboard. 0 Ports (2 rear, 2 via header, 1 Type-A), 4x USB 2. 0 Ports (2 rear, 2 via header), 1x VGA Port, 1x Serial Port (rear), 2x RJ45 LAN Ports, 1x RJ45 Dedicated IPMI LAN Port Form Factor: ATX, 12.

www.ebay.com

 

[Wasmachineman_NL]

This can be good start

AMD EPYC 7551P CPU 32 Cores + Supermicro H11SSL-i Motherboard +4x 32GB 2133P RAM | eBay

AMD EPYC™ 7551P. 1x Supermicro H11SSL-i motherboard. 0 Ports (2 rear, 2 via header, 1 Type-A), 4x USB 2. 0 Ports (2 rear, 2 via header), 1x VGA Port, 1x Serial Port (rear), 2x RJ45 LAN Ports, 1x RJ45 Dedicated IPMI LAN Port Form Factor: ATX, 12.

www.ebay.com

Zen 1 has atrociously bad IPC compared to Zen 3 or Alder Lake though.

 

[Branko]

Zen 1 has atrociously bad IPC compared to Zen 3 or Alder Lake though.

True...but find me one for similar price

 

[RobstarUSA]

Not sure if OP could have got in on the ram deal @Samir posted, but I used that to build a 5950x "server'. Single threaded should be 2.5x (3465/45857) and power usage in general should be lower than intel high-core-count parts.

Add in 128GB ecc udimm from that deal (or if not, they are $140 ea) drop in an Asus/asrock board that supports ecc udimm and should be good to roll.

I'm kind of doing the same thing. I currently have:
3900x/128G for vm severs
5950x/128G on the build bench to replace it.
Nehalem based storage server. The 3900x will become storage server. Nehlaem to the trash.

I bought 4 more dimms from that deal to build another box if needed.

I think both from a "supporting up to 128GB ecc udimm" and "low power" perspective, the current ryzen 3xxx and 5xxx parts are a great value assuming you don't need intel specific cpu extensions.

I've seen 3950x for $300 on ebay, and 3900x at microcenter are $270 now. even 5950x have been under $500 and 5900x @ $370. Great value all around!

If those cpus still eat too much power, even the 65w cpu like 5700x (3347/26538) should be a great improvement from that epyc you have.

Now if you want all 4 servers in 3U, I don't know if that necessarily would work but I'm sure ASRock might have a 1U solution you can maybe buy 3 or 4 of...I'm not up to date on the 1RU barebones offerings these days. I think the ASrock 1U may have the IPMI stuff you are looking for, but perhaps someone else can comment who knows better.

 

[Samir]

Not sure if OP could have got in on the ram deal @Samir posted...

Nehlaem to the trash.

I may have posted that deal, but that was your find dude and you know it.

And don't trash that Nehalem--someone out there will use it as their first server. Or I can add it to my cluster of old working stuff.

 

[altano]

Can you give us some idea of the workflow, like do you build exclusively on one of the servers and run tools etc on the other, or is it all spread evenly among the two boxes?

My server uses are:

• Windows gaming VM (~2 of them) accessed remotely. Casual gaming with friends.
• Fun VMs/containers for various things:
  • Minecraft servers
  • Veeam
  • Home Assistant / Influx DB / Grafana / Prometheus / etc
  • Seafile
  • Plex / Jellyfin (no encoding)
  • Wireguard endpoint for friends
  • etc
• Homelabbing
  • Software I've written and am testing out
  • Always in flux, lots of in & out. The usual suspects (gitea instance, experimenting with zerotier and other networking related applications,
• Programming / software development
  • I run an LXC container in Proxmox that I do all my software development on.
  • I compile/build software on here

Everything is heavily concentrated on one machine right now but mostly because it's more powerful. The only reason I want 2 nodes is to have some availability (e.g. I can move my ZeroNSD VM to the other server while I reboot) and because it's fun to play with a Proxmox cluster. So yes, I could definitely have one fast machine and one slow one as that is my current setup and it's fine.

The only thing I'm bottlenecked on is my personal software development. I'm close to my limit for the gaming VM and the Minecraft servers as I could use more power there but it's fine. But I'm sick of slow software build times and would like to address that for sure.

Anyways, if you're struggling for single-core performance (know how that goes), I'd consider E-series Xeons or something like an ASRock Rack X470/X570 boards with a suitable Ryzen. Either option would support up to 128GB RAM if paired with an appropriate motherboard (IPMI + ECC totally an option with either), and these will generally clock higher than a server CPU at a given $ and power-budget.

I hadn't considered am4/Ryzen. This is a really interesting option and could be very cost effective for the price. Very intriguing suggestion!

True...but find me one for similar price

w.r.t. the Epyc Zen 1 processors: unfortunately the 7551p looks to be the same single core clock speed as the Epyc 3251. Nice price, though

 

[altano]

Some interesting options I have on my candidates list right now:

• $1,551 + RAM - Asrock Rack 1U4LW-X570 (barebones AM4 system) + Ryzen 5950x
• $,1996 + RAM - Asrock Rack 1U4LW-X570/2L2T (10gbe NIC)
• $3,436 - Supermicro AS-1014S-WTRT (Supermicro eStore)
  • EPYC 7313p (16c/32t, 2713 single core, 41884 cpu mark)
  • 128gb RAM (8x16)
  • Upgrades:
    • 7313p => 7443: +$1,045 (24c/48t, 2950 single core, 57490 cpu mark)
    • 128gb => 256gb RAM (8x32): +$608

Things I'm going to keep an eye on:

• Asrock Rack 1U2N2G-B550 - 1U, 2-node AM4 system: marked as "preliminary" on their site and I can't find any google results for
• Ryzen 7000 (7950x?) coming out at the end of this month

 

[Branko]

w.r.t. the Epyc Zen 1 processors: unfortunately the 7551p looks to be the same single core clock speed as the Epyc 3251. Nice price, though

Keep in mind that some compilers can use multithreading, so 64 X 1GHz cores will do job much faster than 1 core at 10GHz...this benchmark list shows nicely that all three factors come into play...number of cores, IPC and frequency...that 5950X surely shows nice value for your use case, but 7551 is not too bad for its price, either

Timed Linux Kernel Compilation Benchmark - OpenBenchmarking.org

openbenchmarking.org

 

[Stephan]

@altano How sure are you that your compilation is optimal already? Build times were also getting on my nerves but then I did some modifications to the Makefiles and now projects build in seconds vs minutes going from a single E3 1275v6 to three 8259CL with distcc.

Edit: For example, used to build kernels on SSD, now with RAM to spare I just copy everything off to /run/user/... and compile in ECC RAM. 3x speedup from that alone.

 

[altano]

@Stephan I’m usually in the JS ecosystem so it’s a lot of poorly optimized garbage. My 8-core CPU actually usually churns through most things very effortlessly but I occasionally hit slowdowns and that’s almost always single core related.

For example I recently started using something with image processing and it was doing it all on a single core, serially. I brought the build time down from ~90s to ~30s by parallelizing it but I can’t always be fixing other people’s code

 

[Stephan]

In that case, wait a little until fall for Zen 4. Improved decoder frontend should speed things up a bit (15%? over Zen 3). Or get a Raptor Lake or Alder Lake, which are AFAIR ECC-capable. But nothing new in hardware will give you a 3x speedup like in your example.

 

[freemarket]

Some interesting options I have on my candidates list right now:

• $1,551 + RAM - Asrock Rack 1U4LW-X570 (barebones AM4 system) + Ryzen 5950x
• $,1996 + RAM - Asrock Rack 1U4LW-X570/2L2T (10gbe NIC)
• $3,436 - Supermicro AS-1014S-WTRT (Supermicro eStore)
  • EPYC 7313p (16c/32t, 2713 single core, 41884 cpu mark)
  • 128gb RAM (8x16)
  • Upgrades:
    • 7313p => 7443: +$1,045 (24c/48t, 2950 single core, 57490 cpu mark)
    • 128gb => 256gb RAM (8x32): +$608

Things I'm going to keep an eye on:

• Asrock Rack 1U2N2G-B550 - 1U, 2-node AM4 system: marked as "preliminary" on their site and I can't find any google results for
• Ryzen 7000 (7950x?) coming out at the end of this month

I just deployed a SMC AS-1014S-WTRT this weekend, got a deal on Newegg with a

• AMD Epyc 7543P 32-Core CPU @ 2800 Mhz
• 128 Gb
• 1 Tb SSD

My only caution is that I had mistakenly understood from Seagate support that the m.2 slot on this system would accept a Gen4 m.2 NVME gum stick and this did not hold out to be the case as the first stick went offline within 24 hrs and was restored by a reboot while the second one went offline yesterday and is still offline, invisible to both superdoctor5 and to seagate's seachest utilities. So still unclear if you can take advantage of their PCIe 4.0 slots with a super fast SSD or NVME stick.

 

[juddle14]

I could get you some E5-2696V4 systems, send me a PM AMD EPYC 3251 vs Intel Xeon E5-2696 v4 @ 2.20GHz [cpubenchmark.net] by PassMark Software

 

[altano]

Thanks everyone for your help. I went with one of BlueFox's systems because, even though it doesn't fit my needs super well, the price was too good to pass up.

 

[altano]

@freemarket that machine is a beast: how good was the deal?