[distcc] Lock before preprocess?

[Martin Pool]

On 24 Aug 2004, Jake McGuire <jake at boom.net> wrote:
> What's the rationale for getting the CPU lock before starting to 
> preprocess?

The rationale is that "3 processes" means "3 processes", not "3
compilers."  It covers linking too.

Preprocessing is quite different to compilation: typically more
IO/bus/VM-bound and pretty light on CPU.  So you could argue that they
have different limits on different machines.  Something with a fast
CPU and slow network filesystem might be relatively slow at
preprocessing.

But as a simple model, you want to run some small multiple of the
number of CPUs on any machine.

> Our situation was admittedly somewhat pathological, but in any case we 
> were seeing preprocessing taking 40%-80% of the time to compile a 
> particular file.  We also had far more jobs scheduled than we had slots 
> in our hosts file, which was exacerbating the slow preprocessing as it 
> looks like distcc attempts to obtain (num_jobs - num_cpus) * num_cpus 
> locks per second.  Which was pushing 5,000 at times, and this over NFS.
> 
> Obviously we've got some internal IT issues to resolve, but is there a 
> reason why getting the lock can't be moved to after the preprocessing 
> step?

I think the real issue here is that the poll-based locking is just not
very efficient when there is contention.  It was designed using file
locks to be simple, portable and robust, and efficient for moderately
low contention.  I'm not surprised it dies when 5000 processes contend
for a couple of locks.  (SysV semaphores do badly when distcc is
abruptly killed, which often happens to compilers.)

A better design would let clients sleep while waiting.  I have some
ideas but suggestions would be welcome.

I'm not sure that -j5000 is really a sensible scenario to support.
Even if it did lock after running the preprocessor then running 5000
copies of cpp would probably melt your machine.  You can try moving it
and see if it does better.

-- 
Martin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://lists.samba.org/archive/distcc/attachments/20040825/a65e2391/attachment.bin