[ccache] Stumbling blocks with ccache and embedded/encapsulated environments

[Martin Pool]

On 11 November 2010 10:56, Christopher Tate <ctate at google.com> wrote:
> I don't want to rain on peoples' parade here, because ccache is a
> great product that has real benefits, but I do want to share some of
> our findings regarding the use of ccache in our very large product --
> we were surprised by them, and you may be as well.  These findings are
> specifically for *large products*.  In our case, the total source code
> file size is on the order of 3 gigabytes (which includes not only
> C/C++ but also Java source files, a couple hundred thousand lines of
> makefiles, etc).  It's the Android mobile phone OS, fwiw: it builds
> something like 1-2 gigabytes of .o files from C/C++ during a full
> build, and does a ton of Java compilation, resource compilation,
> Dalvik compilation, etc as well.

I'd love to know whether you also tried distcc for it, and if so what
happened or what went wrong.  (Obviously it can only help for the
C/C++ phases.)

Your basic point that ccache can only help if you're doing strictly
unnecessary builds is quite correct; eliminating these at the build
tool level may be more elegant or efficient.

> The issue is around VM/file system buffer cache management.  If you're
> using ccache, then you'll effectively be doubling the number of .o
> files that are paged into memory during the course of a build.

I'm just trying to understand how this happens.  Is it that when
ccache misses it writes out an object file both to the cache directory
and into the build directory, and both will be in the buffer cache?
So it's not so much they're paged in, but they are dirtied in memory
and will still be held there.

Perhaps in an ideal world ccache could tell the kernel that it's less
useful to hold the .ccache o file (which probably won't be used until
the next rebuild) than it is to hold the working tree o file (which
may be used soon by the kernel.)  We could try madvise MADV_DONTNEED,
at least to see what happens.

It seems like turning on compression would reduce the effect.

Turning on hardlinking might eliminate it altogether, though that
could have other bad effects.

-- 
Martin