[distcc] SMP volunteers + compression

[Marko Mikulicic]

Martin Pool wrote:
> On  9 Jun 2002, Marko Mikulicic <marko at seul.org> wrote:
> 
> 
>>If I understood well, distcc does lock the volouteer until it the job
>>it is running is completed. I think it should be important to add some 
>>kind of queuing, to please SMP volounteers, and to improve "pipelining" 
>>even on UP machines. I thought of something like DISTCC_HOSTS="a b:2 c 
>>d:4 e:2" where the number after the colon is the queue depth for a given
>>volounteer, which defaults to 1.
> 
> 
> Yes, using SMP machines more efficiently is important.  I would rather
> avoid making each client know how many remote CPUs there are, on the
> principle of (store that information) "once and only once".

Do you mean that distccd should provide this information ?
  Distccd could be started with a cmdline switch which tells him
how many concurrent compilations should be started. I don't
this calling it "processors" is correct, because althrough SMP
systems have the major benefit from it.

> 
> Yes, I think that would help, either
> 
>  - on slow networks (10Mbps, 802.11b)
>  - when there is much more remote than local CPU power
> 
> At low levels of compression, gzip is quite cheap.  Some rough
> measurements show that gzip -z3 cuts .i files by 80%, but uses less
> CPU than running cpp, so it would probably be worthwhile.  I'm
> implementing that now.

I was thinking of something like liblzo, which would scale better
even on faster nets. I notices a traffic of mean 1.5Mb/s with 4 clients
compiling small c++ (with few standard headers) sources. (116 files,
10k lines unpreprocessed, 600k lines preprocessed). This already is
the maximum for 10Mbps lines.

Marko