[ccache] useful test package to use with ccache at the outset in Gentoo.....
michael.fothergill at gmail.com
Sat Dec 30 22:50:00 UTC 2017
On 8 December 2017 at 15:13, Scott Bennett <bennett at sdf.org> wrote:
> Michael Fothergill via ccache <ccache at lists.samba.org> wrote:
> > On 8 December 2017 at 01:35, Scott Bennett via ccache <
> > ccache at lists.samba.org> wrote:
> > > Michael Fothergill via ccache <ccache at lists.samba.org> wrote:
> > >
> > > > I have an amd64 kaveri box with 8GB RAM and run Gentoo stable on it.
> > > >
> > > > I have just installed ccache with 2GB memory allocated to it.
> > >
> > > By that, I assume you have allocated some kind of memory-based
> > > for the cache. Is that a correct understanding?
> > >
> > ?Thanks for your reply and comments. I am assuming that by having the
> > standard command ?
> > ?CCACHE_SIZE="4G" (I have increased the allocation)
> As ccache is installed on my system, I cannot find an environment
> variable of that name documented. Are you sure it is being used? What I
> do find documented are CCACHE_MAX_SIZE and the corresponding ccache.conf
> parameter max_size. If you're going to build large items like libreoffice
> or a LINUX world, you probably ought to have a much larger cache. I also
> found it a good idea to change limit_multiple from its default value of 0.8
> to 0.95 to avoid half-hour-long cleanups in the middle of build runs.
> It is worth noting that, by my understanding of ccache the last time
> I dug into it a bit, ccache actually maintains 16 distinct caches, but the
> max_size and limit_multiple values apply to the total size and usage
> fraction of the aggregate of all 16 caches. When a cleanup occurs, ccache
> chooses one of the 16 caches and begins deleting the least recently used
> entries in it until the total space allocated has been reduced to the
> limit_multiple fraction of the max_size. If it runs out of things to
> delete in that cache while the total allocated remains above that fraction,
> ccache chooses another cache from which to begin deleting entries, and so
> This procedure differs from the one described in the ccache man page and is
> one reason I like to give more space to the cache(s) in order to prevent
> recent entries from disappearing from a cache while other, far less
> used entries remain in the other 15 caches. By having a max_size large
> enough to hold the last several iterations of frequently built items,
> cleanups are more likely to satisfy the limit_multiple by deleting the
> few iterations of updates while sparing the more recently used entries
> in a cache. These days the disk space is cheap enough to give it 10 GB to
> 30 GB without creating any problems for me, so I just do that. Another
> to keep the caches useful is to allocate separate ones for different
> For example, I set up one for building the OS userland and kernel, another
> for building libreoffice, and a third for building everything else. Doing
> this keeps the OS and libreoffice from evicting everything else or each
> prematurely. :-)
> > then memory from the hard drive is being used by default here - I was not
> > trying to use e.g. RAM memory.
> Oh. Okay.
> If your system is used heavily for compiling software, you may see
> performance gains from putting the cache area onto an SSD. Using a system
> memory-based cache is, of course, lightning fast, but the entire cache
> evaporates when the device is deallocated (e.g., during a system shutdown
> failure). I tried using software five- and six-way RAID-0 devices for the
> file system containing my caches for a while, but decided their performance
> was poor. At present I'm using a software CONCAT made of two two-way
> RAID-1's, all on the same kind of hard drives as the earlier setups, and
> setup seems to do very nicely for now. I just have to remember not to run
> updates at the same time as scrubs on the six-way raidz2 that occupies the
> bulk of the same drives. :-) I only scrub that pool about every three to
> weeks, though, so it usually isn't a problem.
> > >
> > > > I have tried some repeat compilations to see if there would be any
> > > > increase.
> > > >
> > > > So far I have not seen much change but I am not skilled enough to
> > > > things yet.
> > >
> > > Your statistics show that slightly more than 45% of your total
> > > compiler invocations (hits/(hits+misses)) were avoided. Did that not
> > > make a dent in your timings?
> > > >
> > > > I tried compiling gcc, glibc and imagemagick but did not see much
> > > > improvement.
> > >
> > > If you run the full build process for gcc, I would not expect
> > > to see much improvement because most of it involves the use of either
> > > a) a temporarily built compiler in a temporary location or b) the
> > > newly built compiler being used for testing, but not yet installed
> > > into the production location on your system.
> > >
> > ?Would cachecc1 perform any better with gcc??
> I don't know what cachecc1 is, so I have no idea. Sorry.
cachecc1 is a gcc cache.
There are some other packages listed under related projects:
e.g. gocache ccontrol and others
Has anyone tried cachecc1 on the site here?
There is no Gentoo package for it so I would need to compile it and install
it by hand.
> > > ImageMagick and GraphicsMagick both should provide useful timings
> > > and ccache statistics. glibc probably would, too, though it's not
> > > nearly as big. I don't know what sort of build procedures Gentoo uses,
> > > but from the FreeBSD ports tree, here are some other good examples of
> > > test cases: math/octave, www/webkit-gtk2, www/webkit-gtk3,
> > > www/webkit2-gtk3, devel/llvm40. Be prepared to wait a long time for
> > > the first compilation of each of the webkits. They are big and slow
> > > to compile and, in the past, have shown instabilities in their build
> > > procedures when parallel make runs were used.
> > >
> > ?I have now compiled qtwebengine, libreoffice and and dramatically
> > the compile time with ccache
> > now I am using the 3.3.4 version of the program ( see here
> > https://forums.gentoo.org/viewtopic-t-1073298.html
> > ? ). I think the earlier version of ccache was not working properly.
> Although some parts of libreoffice remain unaffected by ccache, the
> build as a whole will show radically lower run times. I hope you enjoyed
> seeing how much shorter it was after the first run with ccache. :-)
> > People on the thread have suggested that maybe cachecc1 might not work so
> > well with gcc in practice and I should stick
> > with ccache as is.
> > But you may take a different view. ?
> I must plead ignorance, so no view at all.
> > > YMMV on another OS.
> > > One big savings for me was in running "make buildworld" and "make
> > > buildkernel". buildworld, on my last machine, was taking about six
> > > hours elapsed time for a first run. When running it later after
> > > updating the source tree, the elapsed time was reduced by 2/3 to 3/4,
> > > depending upon the number and sizes of source modules affected by the
> > > updates.
> > >
> > ?These sound like the emerge --ask --update --deep @world command used in
> > Gentoo (and similar versions).?
> buildworld is the make target that oversees the compilation of all of
> the FreeBSD userland. buildkernel is the corresponding target for the
> FreeBSD kernel and all of its loadable modules.
> > ?It's not time to update the system again yet but I think that it could
> > well speed things up as long as the cache size is big enough for a lot of
> > packages (e.g. 200 plus)
> Again, I'd recommend setting up a separate cache area for it. I
> currently allow 10 GB for the libreoffice cache, 16 GB for the OS build
> cache, and 30 GB for everything else. All three of these and
> the build temporary area for ports are physically in the same file system.
> The libreoffice cache and OS build cache space allowances are probably
> bigger than is really helpful, but I have the space, and space not being
> used is space being wasted. FWIW, the cache for building ports averages a
> 50% - 55% hit rate, which adds up to quite a bit of CPU and elapsed time
> saved in the course of a month.
> Scott Bennett, Comm. ASMELG, CFIAG
> * Internet: bennett at sdf.org *xor* bennett at freeshell.org *
> * "A well regulated and disciplined militia, is at all times a good *
> * objection to the introduction of that bane of all free governments *
> * -- a standing army." *
> * -- Gov. John Hancock, New York Journal, 28 January 1790 *
More information about the ccache