Distributed compilation with networked storage

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

victhor393

New Member
Feb 13, 2022
26
11
3
Is there a distributed compiler that can efficiently handle source code stored in a central, networked, storage location, accessible to all worker nodes?

It seems that distcc and icecc are meant to be run on a machine with source code stored locally, which is then distributed as needed to worker nodes.
So using these with NFS is inefficient, as the user's machine would copy the required files from the NFS share, and then send them to the workers.
If the workers have access to the NFS share, then this shouldn't be necessary, but is there any such software that is meant to be run with a setup like this one?

Sure, I could compile everything through the NAS, in which case distcc or icecc would work as intended, but I'd really like to do it from a machine I actually use, as it's likely I'll already have any tools I need installed, it avoids unnecessarily using up storage on the NAS, by avoiding the need to install loads of development tools on it. Not to mention that it's more convenient, no need for remote desktop tools, X11 forwarding, SSH and such.

The idea behind this is that the NAS might have a fat pipe to the network, but the workers might not. If the source code is available through the NAS, it should perform better than if the worker that started the compilation handled the distribution of files between the NAS and other workers.

I tried searching for a solution, but it really seems my case is rather unusual and most people just store source code in their development machine, even if only while it's being actively worked on, hence why no one's thought of it. It makes sense from a performance standpoint, as compiling through a network share is typically going to be slower than local storage, so I understand why I might be an exception here.
 

victhor393

New Member
Feb 13, 2022
26
11
3
Deploying a full CI solution isn't exactly what I had in mind, but it might be worth a look.