[distcc] SMP volunteers + compression

[Martin Pool]

On  9 Jun 2002, Marko Mikulicic <marko at seul.org> wrote:

> If I understood well, distcc does lock the volouteer until it the job
> it is running is completed. I think it should be important to add some 
> kind of queuing, to please SMP volounteers, and to improve "pipelining" 
> even on UP machines. I thought of something like DISTCC_HOSTS="a b:2 c 
> d:4 e:2" where the number after the colon is the queue depth for a given
> volounteer, which defaults to 1.

Yes, using SMP machines more efficiently is important.  I would rather
avoid making each client know how many remote CPUs there are, on the
principle of (store that information) "once and only once".

>  In addition, I was wondering if compression the preprocessed stream
> could help improving performances, expecially when there are many 
> volounteers. The compression should be chosen doing a tradeoff between
> compress ratio and processing needs. (not gzip)

Yes, I think that would help, either

 - on slow networks (10Mbps, 802.11b)
 - when there is much more remote than local CPU power

At low levels of compression, gzip is quite cheap.  Some rough
measurements show that gzip -z3 cuts .i files by 80%, but uses less
CPU than running cpp, so it would probably be worthwhile.  I'm
implementing that now.

-- 
Martin