[ccache] direct mode design bug

[Andrew Stubbs]

On 07/11/12 19:19, Joel Rosdahl wrote:
> It would be nice if ccache were only used and enabled by conscious users
> who have read and understood the documentation, but in reality that
> doesn't happen in many cases. For instance, Linux distributions like
> Fedora install and enable ccache by default (masquerading the system
> compiler), at least when installing the development environment or
> similar. That's not surprising given that ccache works very well for
> most people and that it is advertised as being very safe.

Hmm, I was not aware Fedora did that, but then I don't use Fedora much, 
and when I have Ccache is transparent enough I wouldn't necessarily 
notice. :)

I am aware that Yocto uses it, by default, and certainly their users 
could stumble of this problem, but again, only rarely.

>     A similar issue, albeit not so interesting, perhaps, is what happens
>     when a user changes some part of the toolchain, but does not alter
>     the "gcc" binary. Ccache won't notice a new back-end compiler, a new
>     assembler, a new linker, a new default specs file or anything like
>     that. Chances are that any differences in the output are harmless,
>     but the cached objects are technically invalid.
>
>
> Right. However, isn't the the fact that ccache may be affected by
> toolchain changes much less surprising than the fact that ccache may
> fail to pick up header files correctly?

That's why it's less interesting.

>     [In fact, I have a use-case in which I have multiple users sharing a
>     cache, and I wanted to be able to uniquely identify the same
>     toolchain across all the installations. The mtime etc. varies from
>     machine to machine, as might the exact tool mix, so I have some
>     local patches to do a much deeper hash of the toolchain binaries,
>     and include those in the object hashes. Even then, in the interests
>     of performance, those toolchain IDs are cached according to the
>     location and mtime, so changing the binutils will cause temporarily
>     wrong toolchain hashes. Would you be interested in such a feature
>     upstream?]
>
>
> Perhaps, it depends on how intrusive it is and how toolchain-specific it is.

Basically, it first does the same as CCACHE_COMPILERCHECK=mtime, and 
uses that to look for a <hash>.toolid file in the cache. If the tool-id 
is cached it reads it from that file, and uses that ID to calculate the 
opject hashes as usual. If the tool-id is not cached then it runs "gcc 
-print-prog-name=..." a few times, hashes the binaries it finds, and 
caches the result for next time. CCACHE_COMPILERCHECK=content causes the 
ID to be re-cached, and =none and =<command> are unaltered.

By this means the cached files can be shared across machines with 
toolchains that really are the same (all the way to the bottom) but 
happen to have different installation times being recognised as the 
same, and hashed as the same, but without having to re-hash the binary 
every time.

An interesting side-effect is that binaries cached in 
CCACHE_COMPILERCHECK=mtime mode are now compatible with those cached in 
CCACHE_COMPILERCHECK=content mode, although those cached in the other 
modes remain incompatible.

My implementation is currently GCC specific.

> Not sure about that. I maybe overlook something, but ccache would "only"
> have to follow all #include statements and note all header files that
> don't exist in the include path list. (When #include is used with a
> #defined token for the filename, fall back to the real compiler.) When
> considering a potential cache hit, reject it if any of the header files
> that didn't exist then exist now.

I was thinking of cases like:

#ifdef SOMETHING_NOT_DEFINED
#include "mystery-header.h"
#endif

Presumably you mean that it will note all the *directories* in which a 
particular header file was not found, on the way to finding it?

>         Anybody got other ideas?
>
>
>     Running the compiler with -v prints the header search directories.
>     You could use that to do your own scan.
>
>
> To use the directories from "cpp -v" (plus directories from the command
> line) to do some optimistic validation was my first thought as well, but
> after thinking more about it I came to the conclusion that it wouldn't
> buy much safety because no subdirectories will be checked, and you can't
> tell which subdirectories to check unless you have parsed the #include
> statements. Also, it would trigger many false negatives.

Yes, false negatives would happen, especially if there are include 
directories within the project source tree. :(

The problem is that I've not been able to think of a way that both 
solves your bug, and doesn't have a serious time-impact on either a 
direct-mode lookup, or a cache-miss.

As it happens, I've been thinking of ways to speed up adding things into 
the cache. I've been profiling the code, and found that, on a 
cache-miss, it spends an significant portion of it's runtime between the 
compiler exiting and ccache exiting. It has occurred to me that if we 
were to return the compiler's results to the user straight away, it 
could then fork into the background and spend as much time as it likes 
populating the cache, without slowing the build time noticeably. 
Compilations of the exact same source are unlikely to occur close 
together, so there's no urgent deadline for these.

Given relaxed time constraints, we could certainly do a little more work 
calculating data to store in the manifest file that could then be 
processed lightening fast on a cache lookup.

So, for each include file, we need to know the list of directories it 
could be found in, and which one it was actually found in. This means we 
need to know what names were used in the original code (a user may have 
specified an absolute path), whether they were included with <xxx.h> or 
"xxx.h", and what directories were in the compiler's search path, and be 
aware of #include_next directives.

Knowing the compiler's search path could be done with '-v' every time, 
or we could cache the default ones, and then "know" what the 
command-line parameters mean, or we could cache the search path for each 
set of input options each time.

[Do all the supported toolchains even provide a means to learn the 
search path? If we're getting into ptrace territory then architecture/OS 
specific code will be required.]

Then, at direct-mode cache-lookup time, we do exactly as now, but also 
have a list of locations where stat should return ENOENT.

>     BTW, gcc has an option "--trace-includes" that might be faster than
>     scanning the preprocessor output, although the compiler still has to
>     do all the same work. Like this: "gcc -E hello.c -o /dev/null".
>
>
> How do you use --trace-includes? It doesn't seem to be documented and
> nothing happens when I try it.

Maybe it was introduced recently?

$ gcc --trace-includes -c ~/hello.c -o /dev/null
. /usr/include/stdio.h
.. /usr/include/features.h
... /usr/include/x86_64-linux-gnu/bits/predefs.h
... /usr/include/x86_64-linux-gnu/sys/cdefs.h
.... /usr/include/x86_64-linux-gnu/bits/wordsize.h
... /usr/include/x86_64-linux-gnu/gnu/stubs.h
.... /usr/include/x86_64-linux-gnu/bits/wordsize.h
.... /usr/include/x86_64-linux-gnu/gnu/stubs-64.h
.. /usr/lib/gcc/x86_64-linux-gnu/4.7/include/stddef.h
.. /usr/include/x86_64-linux-gnu/bits/types.h
... /usr/include/x86_64-linux-gnu/bits/wordsize.h
... /usr/include/x86_64-linux-gnu/bits/typesizes.h
.. /usr/include/libio.h
... /usr/include/_G_config.h
.... /usr/lib/gcc/x86_64-linux-gnu/4.7/include/stddef.h
.... /usr/include/wchar.h
... /usr/lib/gcc/x86_64-linux-gnu/4.7/include/stdarg.h
.. /usr/include/x86_64-linux-gnu/bits/stdio_lim.h
.. /usr/include/x86_64-linux-gnu/bits/sys_errlist.h
Multiple include guards may be useful for:
/usr/include/wchar.h
/usr/include/x86_64-linux-gnu/bits/predefs.h
/usr/include/x86_64-linux-gnu/bits/stdio_lim.h
/usr/include/x86_64-linux-gnu/bits/sys_errlist.h
/usr/include/x86_64-linux-gnu/bits/typesizes.h
/usr/include/x86_64-linux-gnu/gnu/stubs-64.h
/usr/include/x86_64-linux-gnu/gnu/stubs.h

$ gcc --version
gcc (Ubuntu/Linaro 4.7.2-2ubuntu1) 4.7.2
Copyright © 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Andrew