Proxmox server build advice needed

[ljlj]

Hi all,

I'd really appreciate some advice on choosing a motherboard & CPU for my 'new' Proxmox server with a budget of around £350/$470 for those two components.
I am happy to buy refurbished gear (I am based in the UK).

I am currently running an old Dell T7610 (2 x E5-2630 v2; 128GB ECC RAM) with 4 x 3.5" 10TB disks (ZFS; 5w idle each).

However, I need something that is:
1. Similar or better performance to existing system
2. Less power hungry at idle (T7610 is 150w idle draw and I'd love to get it down to around 80-ish)
3. Quieter (see then fans below for an idea of what I consider quiet but it's going to have to sit in my living room!)
4. Able to connect 10-12 drives (6-8 HDDs, 4 SSDs)
5. Support for up to 192GB ECC RAM.

I've looked at the Intel E52650L v4 as it has a low TDP, low baseline frequency and 14 cores. Looks to have single-threaded performance not far off my existing system but the additional cores should come in handy for virtualisation. Thoughts?

In the mid-term I want to upgrade to 10G networking (just a matter of power and thermals).

Nice-to-have:
Out of Band management/iKVM (ideally web-based)
Capacity to add a HBA card later either for internal disks and/or a homebrew DAS.


Below is a list of the various components I'll be using for the rest of the build:
Chassis: Logic Case LC-4650-WH (£100)
Chassis fans: Noctua NF-P12 redux-1700 PWM x 4 (£52)
CPU HSF: Noctua NH-U12DX i4 (or Noctua NH-U9DX-i4) (£70)
PSU: be quiet! BN283 Straight Power 11 750W (£130)
RAM: 2 x 32GB ECC modules (just to start with due to budget)

For a single socket configuration I've been looking at two boards but I'd love some other suggestions (including dual-socket)!
Gigabyte MU70-SU0 (circa £140)
Asus Z10PA-U8 (circa £100)

 

[ljlj]

*bump* :-D

 

[nexox]

I've looked at the Intel E52650L v4 as it has a low TDP, low baseline frequency and 14 cores.

The L Xeons from that era confusingly use more power, particularly at idle - they're not binned to run at lower power, they're rejects of dies that were intended to run a higher clocks but can't, and whatever flaws they have also tend to reduce efficiency. I would instead ignore base clock and TDP and look for CPUs with higher Tcase temperature maximum ratings, because those are the dies that are binned for quality, perhaps a non-L E5-2650 V4 (80C Tcase) or even a E5-2680 V4 (86C.) I would also avoid any E5 V4 with more than 2.5MB of cache per core, because that means they have cores present but disabled and in my experience that leads to higher idle power consumption.

That PSU is probably too large and at 80W idle you'll be below 20% load, which means poor efficiency, it is admittedly hard to find really small gold/platinum rated options, but you would probably be better off with a 550W than a 750.

I don't know what prices are like, but I would look at Supermicro motherboards as well, perhaps an X10SRi-F, but any model that starts with 'X10SR' will fit the CPUs you're looking at. I don't think you're going to significantly improve your current idle power with a dual socket board, but they start with X10DR if you want to browse.

 

[ljlj]

@nexox
First of all thank you very much for your time and advice- it was extremely helpful.
By the time of your reply I had already purchased the 2650L v4's and so I have some real world testing to validate against.

Your predictions turned out to be correct. I tested these CPU's in different machine (a Dell T7810) which had two Intel E5-2620 v3's and idled at about 70-75w. Swapping them for the E5-2650L v4 increased idle power consumption to about 80-115w (at first sight not massively more at the lower-end but but a much bigger top-end value). I also noted with interest that although the E5-2650L's seemed to run a lot cooler it was, in fact, because the fans were ramping up much earlier and spinning at higher RPM's.
Under CPU benchmarking (Passmark), the dual Intel E5-2630 v2's used about 239w peak and the dual 2650L v4's used about 246w peak (sysbench has a them at 177w and 197w, respectively).

My question then is this: I am currently sorting CPU's in my spreadsheet by performance and then by Tcase. Should I also still be considering TDP?

I know that comparing TDP's has a lot of issues (I read the forum sticky post) but within a product line like this does it has any predictive efficacy? Would it be useful to have it as my third sorting criterion as it, one presumes, gives some rough indication of worst-case power-consumption?
I feel I may have answered my own question given that the two CPU's I've tested are respectively labelled as 85w and 65w TDP and yet the difference in peak power consumption didn't seem that far apart in the Passmark benchmarks (although sysbench seems to line up nicely enough).

I'm leaning towards your recommendation of the E5-2680 v4 as I would expect better idle power usage based on the Tcase value.

Also thank you for the PSU information as I didn't realise they lost efficiency at the very low-end.

Another interesting thing to come out of the testing was that the E5-2650L's memory performance was worse than the E5-2680 v4 to the tune of about 15%. Naturally the CPU results were better by about 60% (again, Passmark).

I'll include my testing results below just in case that helps anyone else with concrete numbers.
But please let the reader bear in mind that I was not hugely rigorous in my testing. For example, I didn't always set the power governor to "performance" (although all tests did use the same governor) and I didn't control for the room temperature when measuring CPU temperatures, nor did I take time to ensure that the systems were heat saturated before I started testing and so on. Nonetheless, the numbers will have some value because they do represent real-world testing on systems that are not brand new.

## E5-2620 v3

Dual E5-2620 v3 @ 70w idle with no load.
about 100w with basic idle load (k3s)
Passmark @ 246w peak
CPU: 15997
MEM: 2133
sysbench approx 177w

### E2650L

Dual E2650L cpu @80w with no load
about 115w with basic idle load (k3s)
Passmark @ 246w
CPU: 25679 (59C peak; fan 1814 RPM) (+ 60%)
MEM: 1844 (-13.5%)
sysbench approx 197w

 

[nexox]

Should I also still be considering TDP?

TDP, for Intel server CPUs at least, is just there to help select a cooler, some CPUs can't even hit their rated TDP at full load, and as you've seen it doesn't mean much about idle or mid-load power consumption. It does tell you the worst case power consumption but it doesn't tell you how much processing the CPU will actually do when it's using that much power. Generally the V4 Xeons should have better power consumption than the V3s across the board.

I'm leaning towards your recommendation of the E5-2680 v4 as I would expect better idle power usage based on the Tcase value.

The only thing is that the 2680v4 has two more cores than the 2650v4, that could increase idle power consumption, though not always, and of course they're good to have if you can load them up. I've been testing on Skylake Xeons, not really too different from Broadwells, and a 10 core Silver 4114 idles the same as a 14 core Gold 5119T (the T in this generation means higher Tcase rating, 90C vs 78C for the 4114.)

I also noted with interest that although the E5-2650L's seemed to run a lot cooler it was, in fact, because the fans were ramping up much earlier and spinning at higher RPM's.

The fans are potentially a good fraction of the difference in power you noticed, I didn't realize the fan curves adjusted for the rating of the CPU, but I guess they must if the 64C rated CPUs are going to work at all.

 

[sergi0]

best thread ever ! I finally found a quick and easily explanation of the TDP for xeon E5 v3/v4.
I did get a L processor thinking that it will have the smallest TDP at idle considering that it has the smallest TDP. Wrong . I give a look at the Intel ARK list and saw that some CPU are 85+ for Tcase, even 91C for the 2658.
I don't like to do a lot of test and prefer making the "right" choices before, but I will order a couple of this old CPU and make some load/idle tests !

Thanks

 

[louie1961]

This is the new server I just built. I am still trying to decide if it will run Proxmox or TrueNAS. Idles at 32 watts, with the HDDs spun down, 40 watts with them spinning. Its not old workstation/server gear like you seem to be focused on so I thought I would throw this up here for comparison to give you another point of view. It is more performant than the Z640 machine with a e5-2690v3 processor that it is replacing. That old HP Z640 draws 120 watts at idle

AMD Ryzen 5 pro 5650ge
Nactua NH-L12Sx77
Gigabyte 550i aorus pro ax
64 gb of ECC DDR4 ram
4x Samsung SM863a enterprise SSD
2x Seagate Ironwolf ST4000VN006
2x Patriot P210 SSD 256gb (boot drives)
Team Group MP44 M.2 2280
M.2 NVMe to SATA adapter card, ASM1166
Corsair RM650 PSU
Fractal Design Node 304
10Gb dual sfp+ NIC (Intel X520-DA2)

For your requirements I would go with an mATX board and a larger case (Node804?) BTW, my new build is dead silent.

 

[daemoch]

Also thank you for the PSU information as I didn't realise they lost efficiency at the very low-end.

Look for Titanium rated PSUs if you want the low end efficiency to be enforced. Only Titanium stipulates the bottom end. It's the only class I buy any more if I have a choice.

 

[nexox]

Look for Titanium rated PSUs if you want the low end efficiency to be enforced. Only Titanium stipulates the bottom end. It's the only class I buy any more if I have a choice.

If you can find a titanium rating in the same size PSU, sure, but mostly I have only seen them at like 1200W and up, at which point a 600W Platinum would get you basically the same efficiency.

 

[argumentum]

..about one of those MS-01 ? I've got one with a HBA card running TrueNAS CORE on top. Electricity wise should be good.

 

[daemoch]

If you can find a titanium rating in the same size PSU, sure, but mostly I have only seen them at like 1200W and up, at which point a 600W Platinum would get you basically the same efficiency.

They come smaller. There's a good short discussion in this forum thread about it for VERY small loads (sub 40w) and why you don't see really small ones as much.
A 600w Platinum would not necessarily get you similar results. And while considering that, keep in mind that the "80 Plus" rating is only one aspect of power delivery; there's a LOT more that a PSU does (or is expected to do), especially 'good' ones.

I tell people to think of the wattage and 80 Plus ratings as the pay rate you get wherever you work. Its a good standard for comparison between jobs, but pay rate doesn't factor in the benefits package like vacation time, 401k, health insurance, education reimbursement, bonuses, family coverage, etc, or the commute time, which are definitely all factors to consider when making an employment decision.
But as a rule of thumb, the higher the 80 Plus rating is, the more likely you will be to get all the bonus benefits generally. Not always, so watch for that, but usually.

 

[daemoch]

Heres a good spreadsheet to go nuts with if you want. over 1000 PSUs in it.

Cybenetics Labs – PSU Efficiency & Noise Level Certifications - Power Supplies

www.cybenetics.com

 

[nexox]

They come smaller.

Sure, but if you can actually find any of the 600-700W Seasonic models for sale they're waaaay more expensive than a quality 600W Platinum PSU, enough that even with my silly Norcal electricity prices the slight extra efficiency would never pay for itself.

A 600w Platinum would not necessarily get you similar results.

I don't really see how that link says much about the difference between a Titanium 1200W and a Platinum 600W for low loads - I've gone through the Cybernetics charts in the past and for almost all load conditions below ~450W the two are very close, within a couple percent, that's what I call "similar."

 

[daemoch]

Not that you don't have any valid points (you do, especially re availability looking at that 600w fanless unit they list), but thats a bit short sighted I think. You're not considering TCO. Raw random "Efficiency" is just one tiny metric in what a thing costs to purchase, use/service, and dispose of.

You get what you pay for is particularly true of PSUs in my experience. I've seen Platinum rated PSUs that were clearly made just to get the Platinum sticker with basically no circuit protections, crazy ripple/noise, and efficiency drop offs that looked like cliffs (especially once they heated up). They usually have really short warranties for a reason. So assuming you get a standard 5 year warranty (and not 90 days, 1 year, 3 year, etc, which is common), and it actually lasts the 5 years and doesnt burn up anything attached to it, youll need to buy 2 more just to equal that one Titanium's stock (12 year) warranty and guaranteed life span. Or you can milk it out and hope its better than the manufacturer gambled it was going to be. Me? I dont take that gamble. I fix/replace too many of them to take those odds. Manufacturers usually hedge their bets, but not by much. And when a PSU goes, it doesn't go alone.

OP is distinguishing between 80w and 150w and even tracks 5w pre HDD, so this tells me they are looking at really small numbers. "Similar" is subjective and you and I may consider it very differently. OP may have another take on it completely. So I wouldn't discount the difference so casually. A drop off to sub-60% efficiency of a system at an idle that low isn't uncommon at all and if it runs in that state most of the day, thats going to add up pretty fast. Plus, it still CONSUMES that higher input wattage, but now it becomes heat (which is also wear keep in mind) and so the cooling system has to run harder...just to sit and do less. Pay more now for a smaller PSU with a smidge higher efficiency and maybe a passive cooling solution (so dead silent, also a noted desire) and a longer warranty and suddenly that initial price increase seems far more acceptable. Add the cost of a passive CPU cooler and make the whole thing dead silent. Costs more, sure, but now its quieter than your cat on shag carpet.

I do think you have a very valid point in going with a 500P vs a 1200T from your example. But that is kind of where the lines cross. If a smaller T can be found its going to quickly surpass the other options, NM if we look at Gold or even lower. And even a 500P is getting hard to find these days. And thats assuming you loose nothing in the warranty or power conditioning and response (quality) of the units.

Full disclosure, I do this stuff for a living; consumer board level electrical repair all the way up to global distributed datacenter architecture. Ive got a few good client's gaming systems on my service bench right now that had less than optimal PSUs in them and it cost them MB, Ram, GPUs, and/or CPUs in the end, never mind the down time and the obviously burned out PSU. And these weren't no-name PSUs; these are major brands you probably own right now. To reference my earlier job-benefits-analogy, they were good Gold 80 Plus 'pay rates' but poor 'benefits packages'. FTR, I did not design or build these systems! So this kind of question has been discussed with me several times in the last couple weeks.

I've cut clients IT dept bills down just by optimizing their equipment stacks to align better with their actual uses and thermal capabilities, usually by cutting down on what they had but didnt need. The last place I pulled the entire IT room up to the main floor, scaled it back a solid 90% in square footage and stuck it in a centrally located closet with passive vents in the walls, and then completely removed the dedicated AC unit for the space; in the winter they dont use as much heat now either for the surrounding employees. I'm not cheap so highering me isn't going to save anyone money initially, but I can usually pay for myself over the TCO savings I uncover. This could be the same kind of thing - spend a bit more now and reap the long term benefits.

 

[daemoch]

Sure, but if you can actually find any of the 600-700W Seasonic models for sale they're waaaay more expensive than a quality 600W Platinum PSU, enough that even with my silly Norcal electricity prices the slight extra efficiency would never pay for itself.

Could OPs build even use a consumer (ATX) PSU? I didnt look. Most of my initial reply didnt relate to the form factor so I didnt bother. But if its a 'proper' rack server with how swappable PSUs, that would effect the available options a lot! If its only ATX, and ignoring budgets, this might be a great solution.

 

[nexox]

There are plenty of very high quality Platinum rated power supplies, sure all the Titaniums very nicely made (for now,) but the 80 Plus rating doesn't necessarily tell you that much about it, and I never advocated buying the cheapest PSU with the desired sticker on it.

I guess I did forget about the 750W Titanium that be quiet makes, because it can't do two 8 pin CPU power connectors so it wasn't an option for my last build, but most people are probably ok with just one, and it's a reasonable price. Still, over-sizing the PSU is super common and almost always leads to wasted power.

 

[daemoch]

I never advocated buying the cheapest PSU with the desired sticker on it.

Didn't mean to imply you did, sorry. Just for anyone reading this thread, my point was thats not the end-all be-all of measuring standards.

And I whole heartedly agree with you regarding PSU sizing. That or they cut corners in an upgrade and reuse a worn out too-small PSU and then cant understand why they get 'random' crashes.