[distcc] Re: keeping localhost busy

[Jeff]

Martin Pool wrote:
> Maybe you should build more than one library at a time, so that even
> when reaching the end of any one build, there are several jobs that
> can be run?

Due to library dependencies, this isn't always easy. For instance, you might 
have a library which contains Qt ".ui" files that need to have "uic" run on them 
to generate headers that other libraries need to include. I suppose you could 
change your make system to do a "make uic" run before the "make install". 
However I think that part of the beauty of distcc is that with most source code 
you don't have to change a thing.

> You don't need to look at the source.  Just tell me if you can think
> of a better algorithm.  Once you can describe it in English or
> pseudocode translating into C is fairly simple.

Since this problem only happens at the end of a build, I was thinking that 
everything would run normally until all the jobs had been assigned to a machine. 
After that point, as soon as localhost becomes idle give it one of the jobs 
already assigned to one of the remote machines, and then take the work of 
whomever completes it first.

> Timothee suggested killing the job on B and re-running it on
> localhost, but for at least this case it would be wasteful because B
> is as fast as localhost.  For C++ code, transit time is relatively
> small.

My suggestion is different in that you wouldn't kill the remote job. I think the 
other thread already established that estimating how long it will take machine B 
to finish its job is very complex. I think there are probably many people using 
distcc who have a localhost which is relatively fast, but a farm which has 
obsolete, hand-me-down, or donated machines. In that case, I think it's likely 
you'll end up waiting for one or two of your slowest machines at the end of a 
build. In the algorithm I described above, the worst that might happen is that 
you startup a job on localhost, the remote host finishes first, and then you 
have to cancel the local job. But considering the localhost was idle anyway 
(probably just sitting around waiting for the last few objects to create a 
library), this isn't a big deal. The only case I can think of that this might be 
an issue is when your localhost is part of the farm, and other people are 
actively using it.

-- Jeff