[distcc] SMP volunteers + compression
mbp at samba.org
Tue Jun 11 09:58:09 GMT 2002
On 9 Jun 2002, Marko Mikulicic <marko at seul.org> wrote:
> If I understood well, distcc does lock the volouteer until it the job
> it is running is completed. I think it should be important to add some
> kind of queuing, to please SMP volounteers, and to improve "pipelining"
> even on UP machines. I thought of something like DISTCC_HOSTS="a b:2 c
> d:4 e:2" where the number after the colon is the queue depth for a given
> volounteer, which defaults to 1.
Yes, using SMP machines more efficiently is important. I would rather
avoid making each client know how many remote CPUs there are, on the
principle of (store that information) "once and only once".
> In addition, I was wondering if compression the preprocessed stream
> could help improving performances, expecially when there are many
> volounteers. The compression should be chosen doing a tradeoff between
> compress ratio and processing needs. (not gzip)
Yes, I think that would help, either
- on slow networks (10Mbps, 802.11b)
- when there is much more remote than local CPU power
At low levels of compression, gzip is quite cheap. Some rough
measurements show that gzip -z3 cuts .i files by 80%, but uses less
CPU than running cpp, so it would probably be worthwhile. I'm
implementing that now.
More information about the distcc