[ccache] ccache cache in RAM -- bypassing file cache?

[David Coppit]

We're trying to debug a performance difference of 12% speedup on a machine
with 12GB versus a 41% speedup on a similar machine with 48GB of RAM.

We've profiled the build, and believe it to have performance degradation if
it has less than 10GB of ram. (5% slower builds with 10GB of system RAM, 10%
slower builds with 9GB of system RAM.)

Our ccache cache is in memory, on a ramfs mounted this way: sudo mount -t
ramfs -o size=1G,mode=0700 ramfs /mnt/ramfs

Our cache size after the initial warming run is 1GB.

Here's what I'm thinking is happening: The OS is reading from the ramdisk
and then caching the files in memory like it normally would. This is
creating file cache contention with other file I/O in the build, offsetting
our performance gain.

For the machine with plenty of spare RAM, there is no file cache contention
in the OS, and we see much better perf. Does this sound reasonable?

I don't know if there's a way to prevent the OS from caching files on the
ramfs mount, except to modify ccache's open() calls to include O_DIRECT. If
I go that route, do I need to just modify the calls in hash.c, manifest.c,
util.c, and gzlib.c?