Compilation server

orki

New Member
Oct 4, 2021
18
6
3
Longtime lurker, but https://forums.servethehome.com/ind...nt-email-with-room-to-grow.34145/#post-314910 finally impelled me to make an account here. Advice on identifying a machine for a compilation server would be much appreciated.

Use case for a development server:
  • Slow desktop development machine (i5-6400 with 16GB RAM)
  • Compilation times are very slow (about 1-5hrs depending on project, all personal; both memory and CPU limited)
  • Would like a beefier machine for compilation and for LSP server
  • Not particularly space constrained, as basement has plenty of space along with other servers (see below)
  • Development server likely to be used for 10-15 hours only per week (in ~1hr sessions) for personal projects
  • Would like to minimize idle power draw; ideally would go into hibernation or the like when not used
  • Would like at least 64GB of memory
  • Only care about Linux support
  • Price ideally less than 200 USD, located in Illinois

My home network is fairly straightforward and should be lightly loaded and should not have any impact on the choice, but in case it does:
  • Wired 1Gbps connections: Dell 3020 SFF (Opnsense, in the process of switching to VyOS for better CLI support) to HP 2530-24G + 2 Aruba IAP-205 (with redundant Dell 3020 SFF + HP 2530-24G combo as backup in case of component failure)
  • Bunch of Raspberry Pis and HP T620s used as music boxes (via Mopidy)
  • A single NUC + SilverStone RS431U with 4 spinning disks acting as backup server, DLNA server and NFS disk server for music boxes
  • The usual phones/TVs/desktops/laptops but nothing significant (no real video usage beyond Netflix streaming a couple of times a week)
 

RTM

Well-Known Member
Jan 26, 2014
791
291
63
With that budget you will likely be limited to relatively old hardware.

Here is an example, that is only slightly over budget + shipping:

I am not sure how much emphasis you put on the location being Illinois, if you want to pickup something in person, you will have to find a local seller (but absolutely use the link above as a reference for price).

If you do not like the specific server, you should have a look at the seller's ebay store, there's plenty of similarly priced servers.
 
  • Like
Reactions: Patrick

itronin

Well-Known Member
Nov 24, 2018
788
451
63
Denver, Colorado
With that budget you will likely be limited to relatively old hardware.

Here is an example, that is only slightly over budget + shipping:

I am not sure how much emphasis you put on the location being Illinois, if you want to pickup something in person, you will have to find a local seller (but absolutely use the link above as a reference for price).

If you do not like the specific server, you should have a look at the seller's ebay store, there's plenty of similarly priced servers.
the reference above is good for pricing in a general way but I'd be very careful with that specific ebay link/item as you will be limited e5-26xx v1 cpu's due to the motherboard revision.

from the seller's post:
"This is a 1U Supermicro X9DRI-LN4F+ Rev 1.01 System and only works with V1 CPU
Will not work with V2 CPU's."

see this thread.
 

orki

New Member
Oct 4, 2021
18
6
3
Then I should probably double my budget to USD 400. My basic question is whether these decommissioned enterprise rackmount servers are the right approach to my needs, or should I be looking at some other systems?
I don't have a particular preference for a local pickup in Illinois; if shipping turns out to be cheaper or allows for purchasing more powerful systems, that works too.
 

MBastian

Active Member
Jul 17, 2016
158
41
28
Düsseldorf, Germany
Maybe you have the chance to test drive some systems to check what performance you get out of them.

As an example: My Dual E5-2690v2 main system is barely faster than my Ryzen pro 4650G when compiling Openwrt. I checked with different `make -j' flags and also compiled from the ramdisk to rule out disk IO issues.

Enterprise hardware has nice features like redundant PSUs and remote management. But these old E5 v2 servers can surely be temperamental, loud and guzzling energy beasts.
 

BoredSysadmin

Active Member
Mar 2, 2019
693
248
43
I'd say that you first must identify what exactly this usage is in most need of - is it CPU, memory capacity and/or bandwidth, or "simply" storage IOs.
Then you can start to design a solution that fits your needs.

But If I have to go blind, generally faster CPU results in fast code compilation.
You've mentioned your current desktop is an Intel Core i5-6400 - I agree its performance isn't spectacular, at a bit over 5k PassMark, then modern desktop CPU do easily 2-3 times that.
Have you considered upgrading your desktop CPU first? I think (you should doublecheck that) your mother board assuming 200 or 300 series should support 7 gen core CPU (bios update maybe required)
something like this would nearly double the CPU speed:
You should be able to win around $200
I'd also bump up the memory at least to 32GB
 
  • Like
Reactions: MBastian

UhClem

Active Member
Jun 26, 2012
172
75
28
NH, USA
OP, how well have you analyzed your resource utilization, and the constraints you're encountering? (linux has a wealth of mechanisms/tools for this)

Do you know you're swapping?
If so, is it only when/because you're using all your cores? (and, ARE you using all cores??)

[ your box can build a full linux kernel in ~25 min; I'm trying to imagine your "personal project" that requires 1-5 hours ]
Is that LSP thing a bottleneck?
(I'm confident the GNU lang suite is well-written/efficient.)
 
  • Like
Reactions: MBastian and paf

paf

New Member
Sep 21, 2020
16
2
3
Portugal
It all depends where the bottleneck really is. If it is memory size, swapping will occur. It may be cache size. Does compilation speed increases with the number of cores? What about clock frequency?

If it is C/C++ that you are compiling, do you need to recompile it every time?
 
  • Like
Reactions: MBastian

MBastian

Active Member
Jul 17, 2016
158
41
28
Düsseldorf, Germany
As others had already suggested: Check what is your bottleneck first. The imho most easy way ist to utilize sysstat. It is easy to understand and provides a ton of useful metrics. Especially useful is %util from 'sar -d', 'sar -W' for swapping.
Also a look into /proc/cpuinfo or the commands 'cpuinfo' and 'turbostat' might come in handy in case you are limited by some BIOS, thermal or system settings.
If your're lucky you can get away with a cheap upgrade like a faster CPU and/or SSD.
 
  • Like
Reactions: paf

RTM

Well-Known Member
Jan 26, 2014
791
291
63
As the others are being reasonable and suggesting proper investigation before a purchase, I will gladly take it upon me to suggest more unreasonable hardware :D

Here is a suggestion that I think may be hitting what you (OP) are trying to achieve:

It is over the $400 budget.

Platform: Dell Precision T7810 (a tower machine, with support for two E5 Intel v3 or v4 CPU's)
Cost: $280

RAM: 64GB DDR4 RDIMM
Cost: $170

Extra heatsink
Cost: $25

CPU: 2x E5-2697v3 (14 core CPUs)
Cost: $115 each (so $230)

Total: $705

It is probably possible to save some money, by looking around more for better deals.
If you go for something like this, you should validate the design, I do not have a machine like this, so I may likely be forgetting something.

It may make sense to talk to some of the resellers, to see if they can spec out a machine like you want (there is a good chance they have both experience with the hardware and the necessary parts).
 
  • Like
Reactions: BoredSysadmin

BoredSysadmin

Active Member
Mar 2, 2019
693
248
43
I vote for RTM's solution ! The hell with budgets, his build will run circles in a well-multithreaded process application, and for that level of performance believe it or not $700 is really really cheap.
 

orki

New Member
Oct 4, 2021
18
6
3
It all depends where the bottleneck really is. If it is memory size, swapping will occur. It may be cache size. Does compilation speed increases with the number of cores? What about clock frequency?

If it is C/C++ that you are compiling, do you need to recompile it every time?
Indeed this is C++, with a lot of template metaprogramming (I know, I know) in the form of expression templates; a typical translation unit's peak compilation memory use on the order of 15GB (with -O2, a bit larger with -O3) with about 20 minutes of compilation time on a HP DL380 with Xeon Gold 6254 CPU @ 3.10 GHz and 128GB memory. To get exact estimates for cache misses and memory bandwidth, I am running gcc for one of the TUs under valgrind; it has been going on for about 15 hours now; will report final usage when it completes. Any single TU stays within a single core and is affected by the single-threaded performance of that core.
The majority of TUs do get recompiled each time. The code is structured carefully to make a lot of compile-time decisions along with a lot of forced inlining which are fantastic for both size (in spite of the inlining) and runtime performance of the resulting binaries. Because of this, each TU depends on roughly the same large set of common headers, and a change in any of those triggers recompilation.
 

orki

New Member
Oct 4, 2021
18
6
3
OP, how well have you analyzed your resource utilization, and the constraints you're encountering? (linux has a wealth of mechanisms/tools for this)

Do you know you're swapping?
If so, is it only when/because you're using all your cores? (and, ARE you using all cores??)

[ your box can build a full linux kernel in ~25 min; I'm trying to imagine your "personal project" that requires 1-5 hours ]
Is that LSP thing a bottleneck?
(I'm confident the GNU lang suite is well-written/efficient.)
The kernel is a fairly small (but very sexy) project when compiling for a single platform; it uses a lot of pointer indirection (for good reasons) which lets it be generic and be able to compile translation units independently. If the output of the kernel menu's configuration system could be encoded as a type (in the C/C++ sense), then compiling the entire kernel as a single translation unit to maximize the opportunity for the compiler to make global optimizations should make for a leaner and faster kernel. Of course, this can't happen today because compilers really cannot see beyond one pointer indirection at best, and the kernel code is not written with this kind of optimization in mind. If it were written that way, we would generate a family of kernels at compile-time for the platforms/use-cases of interest, which would roughly match what I do in my projects.
LSP is used only during development (for features similar to Intellisense on Visual Studio) but as it is essentially managing an AST of the code in memory as it changes, it too takes a lot of memory, but memory bandwidth is not an issue. My previous comment provides a little more context on the constraints when testing on a (too expensive for my hobby project) HP server to which I have access.

EDIT to answer the other question above: I don't always use all 4 of my cores because it starts swapping and everything slows down to a crawl.
 
Last edited:

orki

New Member
Oct 4, 2021
18
6
3
Maybe you have the chance to test drive some systems to check what performance you get out of them.

As an example: My Dual E5-2690v2 main system is barely faster than my Ryzen pro 4650G when compiling Openwrt. I checked with different `make -j' flags and also compiled from the ramdisk to rule out disk IO issues.

Enterprise hardware has nice features like redundant PSUs and remote management. But these old E5 v2 servers can surely be temperamental, loud and guzzling energy beasts.
Thank you for this comment. You are answering the heart of my question.
 

orki

New Member
Oct 4, 2021
18
6
3
As the others are being reasonable and suggesting proper investigation before a purchase, I will gladly take it upon me to suggest more unreasonable hardware :D

Here is a suggestion that I think may be hitting what you (OP) are trying to achieve:

It is over the $400 budget.

Platform: Dell Precision T7810 (a tower machine, with support for two E5 Intel v3 or v4 CPU's)
Cost: $280

RAM: 64GB DDR4 RDIMM
Cost: $170

Extra heatsink
Cost: $25

CPU: 2x E5-2697v3 (14 core CPUs)
Cost: $115 each (so $230)

Total: $705

It is probably possible to save some money, by looking around more for better deals.
If you go for something like this, you should validate the design, I do not have a machine like this, so I may likely be forgetting something.

It may make sense to talk to some of the resellers, to see if they can spec out a machine like you want (there is a good chance they have both experience with the hardware and the necessary parts).
This is a nice idea. I had not considered workstations, in spite of using a Dell Precision 5610 as my desktop and jump server at work. Is replacing my desktop machine with such a workstation a better idea than a separate compilation server? My (likely flawed) thought process behind a separate server was as follows:
  1. Power consumption for the fancy server would be limited to the times it was doing actual work. My current desktop is adequate for all other purposes and does not seem to be a particular power hog, though I should measure this. Would the workstation's idle power consumption be comparable?
  2. WAF: a loud (at full speed) server would be better tolerated in the basement. I had not considered workstations, but my desktop at work (single Xeon E5-2637 v2 @ 3.50GHz) has not bothered me in terms of noise, though we have white noise generators in the ceilings.
  3. Decommissioned servers (for equivalent performance) would be cheaper than consumer hardware. It appears that I am wrong on this one.
 

cageek

Member
Jun 22, 2018
48
48
18
This is a nice idea. I had not considered workstations...
An older HP z820 with a couple of E5 v2 and as many cores/threads as you can get might be a good option. Maxing the core/thread count is much more important then the CPU speed itself.

You indicate C++, so might I suggest an NVME disk in addition to the ssd, to hold the compilation cache (ccache). It really improved speeds on my large C++ proj. A cheap adapter, and a fast nvme disk (in the fastast PCIe slot possible). You could probably install root on the thing too (just not boot). These old E5 machines don't boot from NVME, but memory is cheap.

If you plan on using suspend/hibernate, it gets better as machines get newer, though. So the Dell 7810 might be better.

Also, I would not spend a premium on the top freq. CPU though. My opinion on E5: v4 vs v3 is kind of a wash, but v3 is better then v2, and you should skip non v2 (v1?). Go by passmark on individual CPUs.
 

RTM

Well-Known Member
Jan 26, 2014
791
291
63
This is a nice idea. I had not considered workstations, in spite of using a Dell Precision 5610 as my desktop and jump server at work. Is replacing my desktop machine with such a workstation a better idea than a separate compilation server?
How you use it is of course up to you.
But what I was thinking was to put the workstation in the basement and access it via SSH or whatever.

BTW most hardware (and almost certainly those workstations) will support wake on LAN, which is what I suggest you use with your setup, it allows you to turn of the server when done, and start it up again by sending a magic packet to it.
Also, I would not spend a premium on the top freq. CPU though. My opinion on E5: v4 vs v3 is kind of a wash, but v3 is better then v2, and you should skip non v2 (v1?). Go by passmark on individual CPUs.
For me the primary purpose for v3/4 over v1/2 is to be able to use DDR4 RAM.
Unless it had to be really cheap (like the $200 budget) I would go with something supporting DDR4
 

cageek

Member
Jun 22, 2018
48
48
18
For me the primary purpose for v3/4 over v1/2 is to be able to use DDR4 RAM.
Unless it had to be really cheap (like the $200 budget) I would go with something supporting DDR4
Yeah, I take back the v2 suggestion. Would definitely go v3/v4 especially after looking at core count availability.
 

orki

New Member
Oct 4, 2021
18
6
3
For me the primary purpose for v3/4 over v1/2 is to be able to use DDR4 RAM.
Unless it had to be really cheap (like the $200 budget) I would go with something supporting DDR4
Yeah, I take back the v2 suggestion. Would definitely go v3/v4 especially after looking at core count availability.
What is the reason for recommending DDR4? Is it because of future proofing against memory bandwidth issues?
 

paf

New Member
Sep 21, 2020
16
2
3
Portugal
Here you have some "memory performance" tools:
Miscellaneous/Tools_Linux_Memory_Perf_Measure.md at master · LucaCanali/Miscellaneous

If your code relies on several levels of pointer indirection, it seems that the processor will always be invalidating the cache contents...
So, main memory bandwidth will be a critical issue.

About the "template metaprogramming", it seems you are using the compiler to do two different things at the same time: write the code that you want and compile it.

Have you considered doing the metaprogramming yourself?