[ccache] Handle concurrency?

[Andrew Stubbs]

On 06/08/15 10:48, Akim Demaille wrote:
> It works very well.  However, especially when running the test suite,
> concurrently, we sometimes compile the same bits multiple times.
> Since some components are really really heavy to compile (two minutes
> for the biggest one, consuming more than a gig of RAM), it's really
> bad to have the machines do it multiple times concurrently.
>
> It seems to me that ccache is in perfect position to avoid this,
> isn't it?  It could detect existing builds and wait for them to
> complete, rather than duplicating them.

I don't believe that ccache is the right tool to solve this.

1. ccache ought to run with as little overhead as possible, and this 
kind of inter-process awareness sounds expensive.

2. distcc is probably better placed.

You should probably check out distcc anyway. It can be chained behind 
ccache (the two tools communicate via environment variables and pipes), 
and has this "job handling" feature already. You'd just need to add some 
logic to recognise matching jobs, according to your definition.

distcc can be used to distribute jobs across a number of machines, but 
it can also be used to queue jobs on the local machine, if you configure 
it like that.

For your use-case it might make more sense to place your modified distcc 
in front of ccache (but beware exec loops!) so that the delayed compile 
is served from cache more straight-forwardly.

Andrew