Proxmox server build advice needed

[ljlj]

Hi all,

I'd really appreciate some advice on choosing a motherboard & CPU for my 'new' Proxmox server with a budget of around £350/$470 for those two components.
I am happy to buy refurbished gear (I am based in the UK).

I am currently running an old Dell T7610 (2 x E5-2630 v2; 128GB ECC RAM) with 4 x 3.5" 10TB disks (ZFS; 5w idle each).

However, I need something that is:
1. Similar or better performance to existing system
2. Less power hungry at idle (T7610 is 150w idle draw and I'd love to get it down to around 80-ish)
3. Quieter (see then fans below for an idea of what I consider quiet but it's going to have to sit in my living room!)
4. Able to connect 10-12 drives (6-8 HDDs, 4 SSDs)
5. Support for up to 192GB ECC RAM.

I've looked at the Intel E52650L v4 as it has a low TDP, low baseline frequency and 14 cores. Looks to have single-threaded performance not far off my existing system but the additional cores should come in handy for virtualisation. Thoughts?

In the mid-term I want to upgrade to 10G networking (just a matter of power and thermals).

Nice-to-have:
Out of Band management/iKVM (ideally web-based)
Capacity to add a HBA card later either for internal disks and/or a homebrew DAS.


Below is a list of the various components I'll be using for the rest of the build:
Chassis: Logic Case LC-4650-WH (£100)
Chassis fans: Noctua NF-P12 redux-1700 PWM x 4 (£52)
CPU HSF: Noctua NH-U12DX i4 (or Noctua NH-U9DX-i4) (£70)
PSU: be quiet! BN283 Straight Power 11 750W (£130)
RAM: 2 x 32GB ECC modules (just to start with due to budget)

For a single socket configuration I've been looking at two boards but I'd love some other suggestions (including dual-socket)!
Gigabyte MU70-SU0 (circa £140)
Asus Z10PA-U8 (circa £100)

 

[ljlj]

*bump* :-D

 

[nexox]

I've looked at the Intel E52650L v4 as it has a low TDP, low baseline frequency and 14 cores.

The L Xeons from that era confusingly use more power, particularly at idle - they're not binned to run at lower power, they're rejects of dies that were intended to run a higher clocks but can't, and whatever flaws they have also tend to reduce efficiency. I would instead ignore base clock and TDP and look for CPUs with higher Tcase temperature maximum ratings, because those are the dies that are binned for quality, perhaps a non-L E5-2650 V4 (80C Tcase) or even a E5-2680 V4 (86C.) I would also avoid any E5 V4 with more than 2.5MB of cache per core, because that means they have cores present but disabled and in my experience that leads to higher idle power consumption.

That PSU is probably too large and at 80W idle you'll be below 20% load, which means poor efficiency, it is admittedly hard to find really small gold/platinum rated options, but you would probably be better off with a 550W than a 750.

I don't know what prices are like, but I would look at Supermicro motherboards as well, perhaps an X10SRi-F, but any model that starts with 'X10SR' will fit the CPUs you're looking at. I don't think you're going to significantly improve your current idle power with a dual socket board, but they start with X10DR if you want to browse.

 

[ljlj]

@nexox
First of all thank you very much for your time and advice- it was extremely helpful.
By the time of your reply I had already purchased the 2650L v4's and so I have some real world testing to validate against.

Your predictions turned out to be correct. I tested these CPU's in different machine (a Dell T7810) which had two Intel E5-2620 v3's and idled at about 70-75w. Swapping them for the E5-2650L v4 increased idle power consumption to about 80-115w (at first sight not massively more at the lower-end but but a much bigger top-end value). I also noted with interest that although the E5-2650L's seemed to run a lot cooler it was, in fact, because the fans were ramping up much earlier and spinning at higher RPM's.
Under CPU benchmarking (Passmark), the dual Intel E5-2630 v2's used about 239w peak and the dual 2650L v4's used about 246w peak (sysbench has a them at 177w and 197w, respectively).

My question then is this: I am currently sorting CPU's in my spreadsheet by performance and then by Tcase. Should I also still be considering TDP?

I know that comparing TDP's has a lot of issues (I read the forum sticky post) but within a product line like this does it has any predictive efficacy? Would it be useful to have it as my third sorting criterion as it, one presumes, gives some rough indication of worst-case power-consumption?
I feel I may have answered my own question given that the two CPU's I've tested are respectively labelled as 85w and 65w TDP and yet the difference in peak power consumption didn't seem that far apart in the Passmark benchmarks (although sysbench seems to line up nicely enough).

I'm leaning towards your recommendation of the E5-2680 v4 as I would expect better idle power usage based on the Tcase value.

Also thank you for the PSU information as I didn't realise they lost efficiency at the very low-end.

Another interesting thing to come out of the testing was that the E5-2650L's memory performance was worse than the E5-2680 v4 to the tune of about 15%. Naturally the CPU results were better by about 60% (again, Passmark).

I'll include my testing results below just in case that helps anyone else with concrete numbers.
But please let the reader bear in mind that I was not hugely rigorous in my testing. For example, I didn't always set the power governor to "performance" (although all tests did use the same governor) and I didn't control for the room temperature when measuring CPU temperatures, nor did I take time to ensure that the systems were heat saturated before I started testing and so on. Nonetheless, the numbers will have some value because they do represent real-world testing on systems that are not brand new.

## E5-2620 v3

Dual E5-2620 v3 @ 70w idle with no load.
about 100w with basic idle load (k3s)
Passmark @ 246w peak
CPU: 15997
MEM: 2133
sysbench approx 177w

### E2650L

Dual E2650L cpu @80w with no load
about 115w with basic idle load (k3s)
Passmark @ 246w
CPU: 25679 (59C peak; fan 1814 RPM) (+ 60%)
MEM: 1844 (-13.5%)
sysbench approx 197w

 

[nexox]

Should I also still be considering TDP?

TDP, for Intel server CPUs at least, is just there to help select a cooler, some CPUs can't even hit their rated TDP at full load, and as you've seen it doesn't mean much about idle or mid-load power consumption. It does tell you the worst case power consumption but it doesn't tell you how much processing the CPU will actually do when it's using that much power. Generally the V4 Xeons should have better power consumption than the V3s across the board.

I'm leaning towards your recommendation of the E5-2680 v4 as I would expect better idle power usage based on the Tcase value.

The only thing is that the 2680v4 has two more cores than the 2650v4, that could increase idle power consumption, though not always, and of course they're good to have if you can load them up. I've been testing on Skylake Xeons, not really too different from Broadwells, and a 10 core Silver 4114 idles the same as a 14 core Gold 5119T (the T in this generation means higher Tcase rating, 90C vs 78C for the 4114.)

I also noted with interest that although the E5-2650L's seemed to run a lot cooler it was, in fact, because the fans were ramping up much earlier and spinning at higher RPM's.

The fans are potentially a good fraction of the difference in power you noticed, I didn't realize the fan curves adjusted for the rating of the CPU, but I guess they must if the 64C rated CPUs are going to work at all.