[ccache] Ccache version 4.0 has been released

[Joel Rosdahl]

Ccache version 4.0 is now available. Release archives can be found here:

    https://ccache.dev/download.html

Summary of major changes:

  - Changed the default cache directory location to follow the XDG base
    directory specification.

  - Changed compression algorithm from Deflate (zlib) to Zstandard, enabled
by
    default.

  - Added functionality for recompressing cache content with a higher
    compression level.

  - Changed hash algorithm from MD4 to BLAKE3.

  - Added checksumming with XXH3 to detect data corruption.

  - Improved cache directory structure.

  - Added support for using file cloning (AKA “reflinks”).

  - Added an experimental “inode cache” for file hashes.

Compatibility notes:

  - The default location of the cache directory has changed to follow the
XDG
    base directory specification (more details below). This means that
scripts
    can no longer assume that the cache directory is “~/.ccache” by default.
    The “CCACHE_DIR” environment variable still overrides the default
location
    just like before.

  - The cache directory structure has changed compared to previous versions
    (more details below). This means that ccache 4.0 will not share cache
    results with earlier versions. It is however safe to run ccache 4.0 and
    earlier versions against the same cache directory: cache bookkeeping,
    statistics and cleanup are backward compatible, with the minor exception
    that some statistics counters incremented by ccache 4.0 won’t be visible
    when running “ccache -s” with an older version.

Changed tooling:

  - CMake is now used instead of Autoconf for configuration and building.

  - A C++11 compiler, a C99 compiler and CMake 3.4.3 or newer are now
required
    to build ccache.

  - Ccache can now be built using Microsoft Visual C++.

Detailed functional changes:

  - All data of a cached result is now stored in a single file called
“result”
    instead of up to seven files. This reduces inode usage and improves data
    locality.

  - Added compression of result and manifest files using the Zstandard
    algorithm. Compression is enabled by default with compression level 1.
This
    makes ccache able to store more data in the cache. Previously
compression
    using Deflate (zlib) was available but disabled by default. Files can be
    recompressed with another compression level later with the
    “-X/--recompress” option described further below.

  - Changed from MD4 to BLAKE3 for hashing input. This improves performance
and
    reduces the risk of hash collisions.

  - Added checksumming of result and manifest files using the XXH3
algorithm to
    detect data corruption.

  - Ccache now follows the XDG base directory specification. This means that
    the default cache directory on Unix systems is “$XDG_CACHE_HOME/ccache”
    (with “~/.cache/ccache” as the fallback if “XDG_CACHE_HOME” is not set)
and
    the configuration file is “$XDG_CONFIG_HOME/ccache/ccache.conf” (with
    “~/.config/ccache/ccache.conf” as the fallback). On macOS, the fallbacks
    are “~/Library/Caches/ccache” and
    “~/Library/Preferences/ccache/ccache.conf”. On Windows, the fallbacks
are
    “%APPDATA%/ccache” and “%APPDATA%/ccache/ccache.conf”. Exception: If the
    legacy “~/.ccache” directory exists, that directory is used as the
default
    cache location and the configuration file is “~/.ccache/ccache.conf”.

  - Cache statistics are now stored in files on cache level 2 to reduce lock
    contention when there are many parallel compilations.

  - An appropriate cache directory level structure is now chosen
automatically.
    The “cache_dir_levels” (“CCACHE_NLEVELS”) configuration option has
    therefore been removed.

  - Added an experimental “inode cache” for file hashes, allowing computed
hash
    values to be reused both within and between builds. The inode cache is
off
    by default but can be enabled by setting “inode_cache”
    (“CCACHE_INODECACHE”) to “true”.

  - Added support for using file cloning (AKA “reflinks”) on Btrfs, XFS and
    APFS to copy data to and from the cache very efficiently.

  - Two measures have been implemented to make the hard link mode safer:
hard-
    linked files are made read-only and inadvertent content changes that
affect
    file size are detected.

  - Added a command line option “-x/--show-compression” which shows
statistics
    about cache compression.

  - Added a command line option “-X/--recompress” which recompresses the
cache
    data with another compression level or makes it uncompressed. If you
choose
    to disable compression by default, or choose to use a compression level
    with a low compression ratio, you can recompress the cache with a higher
    compression level after the build or at another time when there are more
    CPU cycles available, for instance every night. Only files that are
    currently compressed with a different level than the wanted level will
be
    recompressed.

  - Added a command line option “--evict-older-than” which removes cache
    entries older than a certain age.

  - Added a command line option “-d/--directory” which specifies a cache
    directory to operate on. It can be used instead of setting “CCACHE_DIR”
    temporarily.

  - A progress bar has been added to show the progress of time-consuming
    options like “-c/--cleanup”, “-C/--clear”, “--evict-older-than”,
    “-x/--show-compression” and “-X/--recompress”.

  - When supported by the CPU, a SIMD-friendly (using AVX2) algorithm is now
    used to scan input source code for “__DATE__”, “__TIME__” and
    “__TIMESTAMP__” macros. This can decrease the number of CPU cycles for a
    direct cache hit with up to 15% in some cases.

  - Some unnecessary “stat(2)” system calls are now avoided when verifying
    header files.

  - Compiler diagnostic messages are now always cached in color. Ccache then
    strips the color codes on the fly when requested explicitly by a command
    line option or when stderr does not refer to a TTY. This allows IDEs and
    terminals to share cached compilation results.

  - The configuration option “compiler” (“CCACHE_COMPILER”) now always takes
    effect if specified. Previously, the configuration option was only used
    when the compiler specified on the command line was looked up via “PATH”
    (i.e., not when an absolute path was specified).

  - Added optional logging to syslog if “log_file” (“CCACHE_LOGFILE”) is
set to
    “syslog”.

  - The compiler option “-fmodules” is now handled in the “depend mode”. If
    “depend mode” is disabled the option is still considered too hard and
    ccache will fall back to running the compiler.

  - Ccache can now cache compilations with coverage notes (“.gcno” files)
    produced by GCC 9+ in combination with “-fprofile-dir=dir”.

  - “realpath(3)” is no longer used for normalization when computing
relative
    paths. This makes it possible to get cache hits when the source or build
    directory is a symbolic link to an absolute path that includes unstable
    information like build IDs or timestamps.

  - Added an “ignore_options” (“CCACHE_IGNOREOPTIONS”) configuration option
    which makes it possible to exclude compiler options from the hash.

  - Added an “absolute_paths_in_stderr” (“CCACHE_ABSSTDERR”) configuration
    option which makes ccache rewrite absolute paths in compiler warnings
and
    errors to relative.

  - Improved handling of umask. The configured “umask” (“CCACHE_UMASK”) is
now
    only applied to files and directories in the cache directory. Previously
    the umask was applied to all files produced by ccache and the executed
    compiler.

  - Ccache is now able to share cache entries for different object file
names
    when using “-MD” or “-MMD”.

  - Clang’s “-Xclang” (used by CMake for precompiled headers), “-fno-pch-
    timestamp”, “-emit-pch”, “-emit-pth” and “-include-pth” options are now
    understood.

  - Added support for the HIP (“C++ Heterogeneous-Compute Interface for
    Portability”) language.

  - The manifest format now allows for header files larger than 4 GiB.

  - Made it possible to once again cache compilations with “__DATE__” in the
    source code.

  - Added handling of the “__TIMESTAMP__” macro.

  - An absolute input source path is now rewritten to a relative path when
    using “base_dir”.

  - “waitpid” system calls interrupted by a signal are now handled
correctly.

  - Made handling of “.dwo” files and interaction between “-gsplit-dwarf”
and
    other “-g*” options more robust.

  - The “couldn't find compiler” statistics counter is no longer incremented
    when ccache exits with a fatal error.

  - Failure to run a “compiler_check” command is no longer a fatal error.

  - Added command line options “--dump-result” and “--extract-result” for
    inspecting and extracting result files.

  - Added a command line option “--checksum-file” for debugging or
evaluating
    the checksum algorithm.

  - Improved error message for “ccache -o=K=V” (trying to set a
configuration
    option named “=K”).

  - Made timestamps in statistics files Y2038-proof.

  - Removed code for populating a newly created configuration file with max
    cache size and max files values for cache directories created by ccache
    versions older than 3.2 (released 2014).

  - Removed knowledge about a top-level “stats” file created by ccache
versions
    older than 3.1 (released 2010).

Other improvements:

  - Improved help text and documentation of command line options.

  - Improved documentation of the “base_dir” configuration option.

  - Improved documentation of preprocessor and direct modes.

  - Added HTML anchors to configuration options in the manual so that it is
    possible link to a specific option.

  - Tweaked placement of “(readonly)” in output of “ccache -s”.

  - Improved visibility of color output from the test suite.

  - Fixed a problem when running the test suite with Clang without a libgcov
    library available.

  - Fixed test suite problems on macOS.

  - Disabled hardlink tests on AFS since it lacks such support.

  - Disabled read-only tests on file systems that lack such support.

Contributors (in alphabetical order) to this release:

  - Alexander Korsunsky
  - Alexander Lanin
  - Anders F Björklund
  - Arne Hasselbring
  - Breno Guimaraes
  - Chris Burr
  - Cristian Adam
  - Deepak Yadav
  - Enrico Sorichetti
  - Erik Flodin
  - Gregor Jasny
  - Harsh Shandilya
  - Igor Pylypiv
  - Ivan Volnov
  - Joel Rosdahl
  - Loïc Yhuel
  - Luboš Luňák
  - Matt Whitlock
  - Mizuha Himuraki
  - Olle Liljenzin
  - Paul Bunch
  - Paul Fultz II
  - Pavol Sakac
  - Philipp Storz
  - Rafael Kitover
  - Ryan Egesdahl
  - Steffen Dettmer
  - Sumit Jamgade
  - Thomas Otto

Thanks!

-- Joel