[distcc] SMP volunteers + compression
marko at seul.org
Tue Jun 11 11:21:03 GMT 2002
Martin Pool wrote:
> On 9 Jun 2002, Marko Mikulicic <marko at seul.org> wrote:
>>If I understood well, distcc does lock the volouteer until it the job
>>it is running is completed. I think it should be important to add some
>>kind of queuing, to please SMP volounteers, and to improve "pipelining"
>>even on UP machines. I thought of something like DISTCC_HOSTS="a b:2 c
>>d:4 e:2" where the number after the colon is the queue depth for a given
>>volounteer, which defaults to 1.
> Yes, using SMP machines more efficiently is important. I would rather
> avoid making each client know how many remote CPUs there are, on the
> principle of (store that information) "once and only once".
Do you mean that distccd should provide this information ?
Distccd could be started with a cmdline switch which tells him
how many concurrent compilations should be started. I don't
this calling it "processors" is correct, because althrough SMP
systems have the major benefit from it.
> Yes, I think that would help, either
> - on slow networks (10Mbps, 802.11b)
> - when there is much more remote than local CPU power
> At low levels of compression, gzip is quite cheap. Some rough
> measurements show that gzip -z3 cuts .i files by 80%, but uses less
> CPU than running cpp, so it would probably be worthwhile. I'm
> implementing that now.
I was thinking of something like liblzo, which would scale better
even on faster nets. I notices a traffic of mean 1.5Mb/s with 4 clients
compiling small c++ (with few standard headers) sources. (116 files,
10k lines unpreprocessed, 600k lines preprocessed). This already is
the maximum for 10Mbps lines.
More information about the distcc