[ccache] ccacje direct mode without fall back to running the preprocessor

[Andrew Stubbs]

On 10/12/15 03:13, vkr wrote:
> Hello Andew,
>> The answer is "no", because it is necessary to run the preprocessor in
> order to fill the cache and record the file list that direct mode lookups
> use.
>
> What are the consequences if we don't let ccache record the file list, but
> just run the preprocessor and
> do a cache lookup using the preprocessor mode?
>
> WiIl I just have additional overhead when the next direct-look up runs? I
> guess this is the only drawback.

If you don't record the direct-mode meta-data then you can never get a 
direct-mode cache-hit. You will rely on preprocessor mode only, for that 
input anyway.

I understood why you might want to skip the preprocessor mode lookup, or 
not bother caching compiles that will never happen again (read-only 
mode), but this seems like an optimization too far?

If a compile is usefully cachable then you should let ccache go ahead 
and do so. Yes, cache-misses are expensive, and yet the ccache overhead 
makes them even more so (my testing showed about 10% overhead, IIRC), 
but it's all for the sake of the cache-hits on subsequent runs.

I suppose the if you *know* that a direct mode check will always fail 
(modified header, perhaps), but preprocessor mode will succeed, then it 
might make sense to skip the direct mode recording so that you can skip 
the direct mode file comparison on the next run. But, in that case, you 
can just simply disable direct-mode for that file (via environment 
variables in your makefile).

If ccache had a complete set of options then you could optimize for 
every eventuality, I suppose. Maybe its worth doing?

CCACHE_DISABLE_DIRECT_LOOKUP
   - but still write meta-data
   - optimize initial builds
CCACHE_DISABLE_DIRECT_UPDATE
   - but still allow lookup
   - optimize cache-misses when direct mode would be pointless on
     subsequent runs
CCACHE_DISABLE_PREPROCESSOR_LOOKUP
   - direct-mode lookup only
   - optimize cache-misses when we know processor mode will also fail
   - makes the cache read-only
CCACHE_DISABLE_PREPROCESSOR_UPDATE
   - like CCACHE_READONLY but we can still write stats

You really are getting into shaving fractions now though. At this point 
you should seriously consider linking ccache statically; run it with 
callgrind and you can see that a significant proportion of time is spent 
before "main" is even called, loading libraries and such.

Most of the rest of the time is spent doing MD4. I have some ideas how 
to optimize that (by sharing them across runs), but nothing ready to post.

Andrew