[distcc] distcc 0.13 and "Default task limit"

[Ajay Agrawalla]

I also discovered the 4 host scenerio yesterday and had sent an email to
this list.  I checked out the latest code and looks like Martin has updated
the code to get around this problem.

Get the latest code and it should work...  There is not much diffrence in
0.13 and the latest.

HTH

ajay

-----Original Message-----
From: Scott Lystig Fritchie [mailto:nospam at snookles.com]
Sent: Thursday, November 14, 2002 4:40 PM
To: distcc at lists.samba.org
Subject: [distcc] distcc 0.13 and "Default task limit"


Greetings!  I've been using distcc for a couple of months now and have
really found it to be quite valuable.  I started playing with distcc
0.13 today and noticed something a bit peculiar with 0.13's "distcc"
behavior when picking a remote machine that didn't happen with
previous versions.  Then again, my situation is a bit peculiar.

"distcc" seemed to do a good job (with its local locking scheme
mucking around in /tmp) of choosing a DISTCC_HOST to send a job to.
However, that doesn't work if I have several machines running "make".
So, I wanted to insert a TCP proxy in between my "make" build machines
and my back-end "distccd" machines.

I was dissatisfied with the TCP proxies that I could find in Open
Source Land.  Most of them were written for application-level firewall
or HTTP proxying use.  Those that load-balanced did simple round-robin
or some variation.  My back-end machines vary quite a bit in terms of
CPU speed: I didn't want the proxy choosing a slow back-end when a
faster one was available...

... so I wrote my own TCP proxy.  It chooses the fastest back-end
machine that happens to be idle.  It nicely deals with multiple-CPU
back-end boxes.  It's written in Erlang, if that interests you or
bothers you.  If anyone is interested in its code, I can see if my
employer would mind letting me release it under free/free'ish license.
{shrug}

Anyhow, I'm leading up to my newly-discovered problem.  I just started
using 0.13 today.  I did the following:

    % setenv DISTCC_HOSTS "proxy proxy proxy proxy proxy proxy proxy proxy
proxy proxy proxy proxy proxy proxy proxy proxy proxy"
    % make -j17 blahblah

And I discovered that the proxy only had 4 jobs active at one time.
The "make" box is an AMD XP1600+, and even though the distcc + cpp
overhead is pretty low, this machine can't actually get 17 going very
often.  But 8-12 is no problem.  Seeing only 4 was a big surprise.

I found something in 0.13's src/hosts.c's dcc_parse_hosts() function:

        /* Default task limit */
        curr->n_slots = 4;

This, and later usage of n_slots elsewhere, appears to limit the
number of jobs that any single host can perform.

I've simply fallen back to using 0.12.  I suppose I could add a few IP
aliases to the proxy box and use DISTCC_HOSTS="proxy1 proxy1 proxy1
proxy1 proxy2 proxy2 proxy2 proxy2 proxy3 ...", but I don't think I'll
go there quite yet.

I assume that this "default task limit" is a feature that isn't
implemented fully yet?

-Scott
_______________________________________________
distcc mailing list
distcc at lists.samba.org
http://lists.samba.org/cgi-bin/mailman/listinfo/distcc
-------------- next part --------------
HTML attachment scrubbed and removed