[distcc] Multiple people running distcc at once, using one DISTCC_DIR

[Daniel Kegel]

Victor Norman wrote:
> All,
> 
> I'm trying to roll-out the use of distcc here in our organization among our 70
> or so software engineers (see previous post "My big plans for distcc").  I'm
> running into a few issues with multiple engineers doing compiles at once.  Here
> are the specs: distcc2.16 on sparc-sun-solaris2.7 and linux.  All engineers are
> going to use one DISTCC_DIR (which is /us/norman/.distcc for now).  So, the
> same hosts file and the same lock and state directories will be used for
> everyone.  This is necessary (see previous posting) so that a single server
> that can handle n compilations is not hit with n * m compilations, where m is
> the number of engineers doing a compile at that time.

No, it's not quite neccessary.  If you apply the patch
http://kegel.com/crosstool/crosstool-0.28-rc34/patches/distcc-2.16/randomhost.patch
and add --randomize at the top of the hosts file,
the clients will randomize the hosts list on startup.  This
will greatly reduce the effect you're worried about.
Have you tried this yet?

> Here are the problems we are noticing:
> 
> 1. We've noticed that the lock files are not cleaned up when a compilation is
> done (i.e., when an instance of distcc finishes).  For example, in
> /us/norman/.distcc/lock, I see 4 files: cpu_tcp_swbuild-linux1_3632_{0,1,2,3}. 
> And, my compilation is done.  I thought those files should have been removed.
> 
> The problem with this is that each of those files is owned by norman (me), and
> has -rw-r--r-- permissions.  So, when my friend, pjayamoh, tries to compile and
> distcc picks swbuild-linux1 as the host to compile on, it will fail because
> distcc won't be able to open/remove/replace the lock file.
> 
> Is this just a matter of changing our umasks?  Or is this a real problem?
> 
> 2. Also, when pjayamoh cannot open that file, distcc falls back immediately to
> trying to compile locally.  That's fine, but I would have thought that it would
> have tried some other slots and/or hosts first, which it doesn't seem to be
> doing (I'm not sure about this one, though -- correct me if I'm wrong).

I don't think you should be sharing a .distcc directory between
multiple users.  It's not a common configuration, and there are
sure to be problems.
- Dan