[ccache] Using a shared ccache in cmake environment (linux)

[Joel Rosdahl]

On Tue, 17 Mar 2020 at 10:06, Steffen Dettmer via ccache
<ccache at lists.samba.org> wrote:
> As workaround for a special unrelated issue currently we redefine
> __FILE__ (and try to remove that redefinition). I understand that
> ccache still works thanks to CCACHE_BASEDIR even for __FILE__ usage
> inside files. Is that correct?

Yes, if basedir is a prefix of the the source file path then __FILE__
will expand to a relative path since ccache passes a relative source
file path to the compiler.

> I understood that CCACHE_SLOPPINESS=file_macro means that cache
> results may be used even if __FILE__ is different, i.e. using a
> __FILE__ from another user (fine for our usecases), is this correct?

That used to be the case, but the file_macro sloppiness was removed in
3.7.6; see <https://ccache.dev/releasenotes.html#_ccache_3_7_6>.

> How to find a reasonable max_size? For now I just arbitrarily picked
> 25 GB (approximately the build tree size) and I never saw it "full"
> according to ccache -s.

"cache size" will never reach "max cache size", so that is not a good
indicator of whether "max cache size" is enough. See
<https://ccache.dev/manual/3.7.8.html#_automatic_cleanup> for details on
how automatic cache cleanup works. The TLDR is that "cache size" will
stay around 90% (assuming limit_multiple is the default 0.8) of "max
cache size" in steady state. This is because each of the 16
subdirectories will be between 80% and 100% full with uniform
probability.

Instead you can have a look at "cleanups performed". Each time a
subdirectory gets full that counter will increase by 1.

> On build servers we usually run "make -j 25" (24 cores). Often,
> several such jobs are running by different users (and Jenkins;
> sometimes 400 compiler processes or even more). I assume ccache of
> course safely handles parallel invocation, is this correct?

Yes, assuming that ccache's locking works on the filesystem in question.
(Locking is only used for the stats files; the actual cache data files
are handled via atomic renames.) The cache files can however get corrupt
if a server crashes, depending on how writeback/journaling of filesystem
metadata is configured.

Especially with network caches it might be a good idea to disable
automatic cleanup and instead perform explicit cleanup periodically on
one server, preferably the server that hosts the filesystem. That way
the cleanup won't happen over slow network and several clients won't
compete to clean up. One way of doing this is to set an unlimited cache
size and then run something like "CCACHE_MAXSIZE=25G ccache -c"
periodically the server.

> We have some team mates that have slow (old) laptops only. They
> benefit from using a network shared ccache. Technically, they "mount
> -t cifs" the cache_dir (NFS is firewalled unfortunately). We have
> different Ubuntu/Mint/Debian/Devuan machines, but exactly the same
> compilers (own toolchains).
>
> Is sharing via CIFS possibly at all or could it have bad effects?

Don't know, but I wouldn't be surprised if ccache's locking doesn't work
properly with SMB/CIFS. Locking is based on creating symlinks atomically
and I guess that doesn't translate well to Windows filesystems.

> One issue that occures from time to time is that the ccache -s stats
> become zero (all values expect max cache size are 0). I first didn't
> notice because stats are shared so I assume someone zeroed the stats,
> but with alternate directories we found that it sometimes happens
> without ccache -s. "du -hs $CCACHE_DIR" still shows gigabytes used. We
> didn't find a cause yet, but several candidates exist.

Sounds like the stats file locking fails...

> Can ccache be used on CIFS?

Answered above.

> Are cache and/or stats version dependent?

The cache data and stats files are intended to be backward and forward
compatible from ccache 3.2.

> A few times we noticed that ccache -s reports few GB size but "du -hs"
> reports 40 or 50 GB, although "max_size = 25.0G". Is this expected?

No. But if stats file locking doesn't work then this could happen.

> I'm also still facing scmake issues (using "physical" and "logical" in
> several mixed combinations). Complex topic.

What is scmake?

-- Joel