[ccache] Radically different ccache performance on Linux vs OSX

[Christopher Tate]

We have a large cross-compilation environment that is essentially
identical on both Linux and MacOS X hosts.  The build system uses
ccache to try to reduce redundant recompilation, but we're seeing
radically different outcomes on OSX versus Linux hosts.  We're still
using ccache 2.4.

On Linux hosts, the aggregate performance seems to be roughly a 3:1
cache hit-to-miss ratio.  However, on OSX, ccache seems to be
providing almost no benefit:  we see something like a 1:15 cache
hit-to-miss ratio -- at which point we might as well not be using
ccache at all.

I've been looking into this a bit with 3.0 (actually tip of tree) on
OSX.  Using ccache 3 [in its default mode of direct mode then fallback
to preprocessor], 90%+ of the misses that I see in the log are failure
to find the manifest file at all, and I'm seeing a similar very low
hit rate overall.  Right now I'm doing some tests around using the new
CCACHE_SLOPPINESS=file_macro tweak, and initial results are
promising... but that begs the question of why the results on OSX with
2.4 are so poor in general.

The obvious guess is that there's *something* about our cross-build
chain and environment that differs from the Linux-hosted versions just
enough to be upsetting ccache.  Any tips on what I should go looking
for?

Thanks!

--
chris tate
android framework engineer