[ccache] Stumbling blocks with ccache and embedded/encapsulated environments

Christopher Tate ctate at google.com
Thu Dec 2 09:42:22 MST 2010

On Wed, Dec 1, 2010 at 9:00 PM, Martin Pool <mbp at canonical.com> wrote:
> On 11 November 2010 10:56, Christopher Tate <ctate at google.com> wrote:
>> I don't want to rain on peoples' parade here, because ccache is a
>> great product that has real benefits, but I do want to share some of
>> our findings regarding the use of ccache in our very large product --
>> we were surprised by them, and you may be as well.  These findings are
>> specifically for *large products*.  In our case, the total source code
>> file size is on the order of 3 gigabytes (which includes not only
>> C/C++ but also Java source files, a couple hundred thousand lines of
>> makefiles, etc).  It's the Android mobile phone OS, fwiw: it builds
>> something like 1-2 gigabytes of .o files from C/C++ during a full
>> build, and does a ton of Java compilation, resource compilation,
>> Dalvik compilation, etc as well.
> I'd love to know whether you also tried distcc for it, and if so what
> happened or what went wrong.  (Obviously it can only help for the
> C/C++ phases.)

distcc can certainly help a great deal.  For us, it's a bit
problematic to use because more than half of our total build is
non-C/C++ that depends on the C/C++ targets [e.g. Java-language
modules that have partially native implementations], plus we have a
highly heterogeneous set of build machines: both Mac hosts and Linux,
not all the same distro of Linux, etc.  The inclusion of Macs in
particular makes distcc more of a pain to get up and running cleanly.

>> The issue is around VM/file system buffer cache management.  If you're
>> using ccache, then you'll effectively be doubling the number of .o
>> files that are paged into memory during the course of a build.
> I'm just trying to understand how this happens.  Is it that when
> ccache misses it writes out an object file both to the cache directory
> and into the build directory, and both will be in the buffer cache?
> So it's not so much they're paged in, but they are dirtied in memory
> and will still be held there.

Even on a ccache *hit* both copies of the .o file wind up occupying
buffer cache space, because the ccached .o is read from disk [paging
it in] in order to write the .o file to the build output directory.
On a ccache miss the copy runs the other direction but you still wind
up with both sets of pages in the buffer cache.

> It seems like turning on compression would reduce the effect.

At the expense of the extra cpu time, sure.  That might be a decent
tradeoff; modern cpus are getting quite fast relative to I/O.

> Turning on hardlinking might eliminate it altogether, though that
> could have other bad effects.

Right.  We haven't tried pursuing this because for other reasons the
marginal returns are still pretty low, and tinkering with the build
system is fraught with peril.  :)

christopher tate
android framework engineer

More information about the ccache mailing list