[ccache] Optimizing MD4

[Andrew Stubbs]

On 11/12/15 16:28, Anders Björklund wrote:
>> Each invocation of ccache then connects to that socket and asks the
>> daemon to do the MD4 scan on its behalf. The daemon checks the mtime on
>> the file and serves the MD4 from its memory cache if nothing has
>> changed. The stat call could probably be optimized away if the cache is
>> very fresh (<1s?)
>
> So basically something similar to the "sloppiness" file_stat_matches ?
> Compare size/mtime/ctime, rather than rehashing the content of a file.

Yes, except that it *has* been checked this session. This is a 
significant difference when you try to share a cache between two 
machines, or two users, or just simply did some git foolery between builds.

Good point though: for a lone user doing nothing exciting, 
CCACHE_SLOPPINESS=file_stat_matches does nearly the same thing with 
considerably less effort.

>> You could, in principle, use this communication to limit how many
>> cache-miss compilations are permitted to run in parallel, and therefore
>> run "make -j" for maximum parallelism without fear of melting your memory.
>
> Something like that is what I meant with a new "prefix" wrapper for cpp.
> Similar to the current wrapper for cc, which does that with e.g. distcc.

Yes. You have to be careful with how many executables you have though: 
the dynamic linker is slooooow.

>> Unfortunately, I've moved on to other projects and don't have much time
>> to work on this stuff any more.
>
> Thank you for your ideas. Will check the code out too if I get the time.
> It seems that there are some opportunities left for faster manifest/cpp.

If you decide to go with the persistent background task approach, then 
there are a few ideas in there to pluck out, but the overall 
architecture had other goals, so maybe it's not that great a starting 
point. :-(

Good luck.

Andrew