Looking for a CPU + Mobo combo for a home lab

[user949]

Hi Guys,

I am looking for a mobo + CPU combination for my home lab. Currently, I have 2x monster ESXi servers, each with 256GB RAM and 12+ cores. I'd like to replace that with a small cluster of 3x (or more) ESXi hosts so that if there is a power outage I can shut down all of the non-essential VMs and hosts and have a longer runtime on my UPS.

Normally I would buy a used Dell/HP from EBay, but I've been burned too many times with no further updates to BIOS or the dreaded bricked iDRAC! I'd made up my mind I want to do a custom build.

I am looking for a mobo + CPU combination that can give me at least 64GB on each node. I'd prefer to run this in a 2U case, but the noise factor is most important, so if I must go to 4U to cut down on noise, that is something I am willing to do.

My main requirements are as follows:

• At least 64GB per node.
• Ideally, the Mobo would support CPUs ranging from 8-core to 16-core (or more) in case I later find a deal CPU deal on EBay.
• Silent/Very quite idle.
• Datastores will be 2x SSD/NVMe drives and 2x SATA Hard Drives.
• IPMI

If you were building the above, what would you go with?

 

[Sean Ho]

64GB and 8-core is no problem; 1151-2 (e.g., X11SCA-F) can do that with very low power. 16 cores is harder, perhaps 3647 (X11SPL-F) or AM4 (only the two ASRock D4U boards)?

 

[user949]

Hi Sean,

Thanks for the response. I think I am going to go with the Intel Scalable processors, per your suggestion. I feel as though I'm more likely to find better deals on EBay if/when I want to upgrade the processor at a later date/time.

What would you recommend for CPU cooling?

 

[i386]

What's the budget?
(Enough for epyc systems?)

If you were building the above, what would you go with?

a supermicro 745 chassis:
- 745 for redundant psus (or 1psu + 1bbu)
- supports eatx mainboards
- standard backplanes can be replaced with a backplane that has 4 u.2 connectors for nvme ssds, (4 bays available for more storage)
- different fans available (eg 2.8k rpm sq fans or 9.4k rpm fans for dual 280w tdp epycs/xeons )
- airshroud (in combination with active 2u heatsinks the highest cooling capacity!)
- supermicro quality

 

[Rand__]

I am looking for a mobo + CPU combination for my home lab. Currently, I have 2x monster ESXi servers, each with 256GB RAM and 12+ cores. I'd like to replace that with a small cluster of 3x (or more) ESXi hosts so that if there is a power outage I can shut

Ideally, the Mobo would support CPUs ranging from 8-core to 16-core (or more)

Are you sure you need so many cores? Just 'cause u have them now does not mean u use them... Also if you go down to E3's u get significant higher frequency that might offset some extra cores...

Not trying to sway you, just making u think

 

[user949]

Are you sure you need so many cores? Just 'cause u have them now does not mean u use them... Also if you go down to E3's u get significant higher frequency that might offset some extra cores...

Not trying to sway you, just making u think

They Rand__ I appreciate the suggestions. Please keep them coming!

I was thinking more about VMware and their recommendations vis-a-vis the number of vCPUs not exceeding the number of actual cores in the host. How strict do I need to be with myself on matching the number of vCPUs against the host's core count?

If this influences your answer, the hosts are for 3x environments: a "PROD" environment, lightly used DCs and a few utility servers, as well as always-on "TEST" and "DEV" environments. The TEST/DEV environments each host Exchange servers, DCs, ADFS and a few other services meant to replicate the environments we develop software within (for clients, for example).

To be perfectly honest, the performance of these environments is of little consequence, and the main thing is that I need to be able to cram many of them into a cluster of quiet hosts.

If I'm to stick to VMware's 1vCPU=1xCORE recommendation, my current VM requirements need 36 cores; ergo, the reason for the 3x 12-core cluster. If I can half that core count and more densely pack my VMs, that would be excellent because I want to do patching/maintenance on one of the hosts without shutting down VMs.

 

[Markess]

I was thinking more about VMware and their recommendations vis-a-vis the number of vCPUs not exceeding the number of actual cores in the host. How strict do I need to be with myself on matching the number of vCPUs against the host's core count?

Now I'm curious. I thought VMware played nice with hyperthreating, doesn't it? For example: If you have a 12 Core CPU with hyperthreading, then you can assign 24 Cores to VMs. At least I thought that was the way it worked. Or...maybe I've been screwing up my VMs on that machine all along?

Edit: Assuming you have a hyperthreaded CPU of course. But,I think almost everything that isn't ARM etc. is these days.

 

[Rand__]

If I'm to stick to VMware's 1vCPU=1xCORE recommendation,

*cough*

This is a 12 Core/24 Threads box... I had to search for cores in the columns to see the actual assigned ones, I *never* look at that - Host CPU is the measurement i observe:



Which tells me I have some spare cycles (this is a Xeon W-3235 CPU @ 3.30GH)
Note this is what I currently run instead of a 4node vSan cluster - the only downside is no redundancy at all...

Just one comment - please note that over-committing cores can lead to latency since the vm waits until the assigned # of cores is usable and these get blocked... so 4 core vm on a 8core box may have to wait... which is why high frequency helps a lot there.

Edit: You see that there are a bunch of idle vms in there -> Host CPU is negligible - those perform background activities (ie non interactive), so a slight delay is no issue. I also run a Horizon client vm on this box without issues.

Edit2: I ran more or less the same vms on a quad E3-1245v6 cluster before I went on a tik cycle (more performance) ... (which is now back to tok (less power) which is why i dont run 4 of the Xeon-Ws yet which sit powered off in the rack

 

[Sean Ho]

VMWare can certainly overcommit CPU; 3:1 or 4:1 vCPU : pCPU ratio is very common. E3 opens up a lot more (and more affordable) options for you, especially if the workload is light. Each VM will still want at least 2GB RAM, preferably more.

 

[user949]

OK. So now you guys have taken me in a totally different direction. One that might save me a bit of money, so thank you!

Below are the CPUs I have at the moment if I were to savage existing HP/Dell systems:

- 2x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
- 2x Intel Xeon 8-Cores E5-2670 2.60 GHz 20M Cache 8 GT/s QPI
- 2x Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz

Since I have a few E5 processors, should I just savage these CPUs and get compatible mobos from Supermicro?

 

[Rand__]

If you plan to run the boxes for a couple of years I would refresh them... these get long in the tooth

 

[user949]

So take the CPUs out and get new mobos?

 

[Rand__]

Cheap e3's v5/6+ or e5's v3/4 would be my recommendation.

 

[Sean Ho]

Below are the CPUs I have at the moment if I were to savage existing HP/Dell systems:

- 2x Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
- 2x Intel Xeon 8-Cores E5-2670 2.60 GHz 20M Cache 8 GT/s QPI
- 2x Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz

Since I have a few E5 processors, should I just savage these CPUs and get compatible mobos from Supermicro?

Your OP mentioned long runtime on UPS, meaning low power draw under normal workloads. These CPUs, while capable and dirt cheap, are the antithesis of that.