[ccache] Caching failed compilations

Anders Björklund anders at itension.se
Tue Jul 7 21:37:08 CEST 2015

Hi Joel and all!

I also found the idea of storing failures interesting, and made a quick sample implementation earlier:


Feature is enabled by setting $CCACHE_STOREFAILURES

It does store the status as a separate file, but on the other hand there is no object file stored for failures.
One could look for a object file (success) *before* looking for a status file (failure), to cut down on stat's ?

My biggest fear is that it will store I/O errors and whatnot, with no "easy" way to rebuild (needs a recache)
So I made it opt-in, rather than default. So far, it seems that almost everything in cc returns exit code "1"

I was investigating cutting down on the number of files, and stored everything in a LMDB* database instead...
The biggest downside is that adding new files to the cache (and cleaning) now becomes much more involved.

* https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database

We already had a compact format developed for the memcache extension* (that is also proving to be very useful).
It does make "extending" the format harder, but I suppose one could use the "version" in the CCH1 header for that ?

* https://github.com/tardyp/ccache/commit/33852da77f54c9227cb90e013e1bb186a7d315c2

I am hesitant to replace the default (files), but see great potential when combining memcached with distcc.
It opens up for sharing a secondary cache between *several* servers, but without having to do a recompile.

So we use it like: ccache <-> memcached <-> distcc

Will clean up database/memcached for public testing...

It is possible to convert between the different cache formats, since the actual files inside are all the same.
A simple conversion script (in python) is provided. A little slow if the cache is in use, but otherwise it's OK.


Från: ccache-bounces at lists.samba.org [ccache-bounces at lists.samba.org] för Andrew Stubbs [ams at codesourcery.com]
Skickat: den 7 juli 2015 10:58
Till: Joel Rosdahl
Kopia: Akim Demaille; ccache list
Ämne: Re: [ccache] Caching failed compilations

On 06/07/15 21:44, Joel Rosdahl wrote:
> That sounds like a reasonable idea, but I have occasionally seen empty
> object files in large and busy caches (it could be due to filesystem
> failure, hardware failure or hard system reset), so I'm afraid that
> using zero-length object files won't work out in practice. See also
> https://bugzilla.samba.org/show_bug.cgi?id=9972. But maybe writing some
> special content to the object file would be OK?

OK, fair enough, but I'd say that once you've opened the file and
checked the magic data then you've already killed performance. How about
a magic length that can be observed in the stat data?

A failure can be confirmed by a read, if and only if the length matches,
but a compile success will remain on the quick path.

A cache-hit for a compile failure need not be the *most* efficient code
path; it will likely end the build process. As long as it's faster than
the "slow" compile failures the OP cares about then all is well.

>     Sorry, I don't see any advantage in this scheme. You might save a
>     few bytes of disk space, and maybe a few inodes, but I've not seen
>     any evidence that those are a problem. You'll also add extra file
>     copies to every cache miss, and those are already expensive enough.
> My primary motivation for considering the mentioned scheme is to reduce
> disk seeks, not disk space. If you have a cold disk cache (on a rotating
> device), every new i-node that needs to be visited potentially/likely
> needs a new disk seek, which is slow. If all parts of the result are
> stored in one contiguous file, it should likely be quicker to retrieve.
> But as mentioned earlier, I have no data to back up this theory yet.

My understanding is that when a disk read occurs the kernel reads the
entire page into the memory cache. Subsequent inode reads will likely
hit that cache, so reading two inodes is nearly as cheep as reading one.
The system call overhead is constant, however.

> A secondary motivation for the scheme is that various code paths in
> ccache need to handle multiple files for a single result. There can now
> be between two (stderr, object) and six (stderr, object, dependency,
> coverage, diagnostics, split dwarf) files for each cached result. If one
> of those files is missing, then the result should be invalid. This is
> quite painful and there are most likely some lurking bugs related to this.

OK, that's quite a lot of files. Hopefully it does not look for a file
unless it really ought to be there? I worry that you'll hurt the common
case (just two files) in order to help the uncommon case, and that that
is already about as good as it can be (especially with hard-links).

> A third motivation is that it would be easier to include a check sum of
> the cached data to detect corruption so that ccache won't repeatedly
> deliver a bad object file (due to hardware error or whatnot).

Any checksum had better be very fast. Profiling ccache already shows
that it spends more time doing MD4 than anything else.

ccache mailing list
ccache at lists.samba.org

More information about the ccache mailing list