[ccache] ccache: FATAL: Could not create xxx.o.tmp.stdout.yyy.zzz (permission denied?)

[Lalit Chhabra]

The tmp file contains the machine name and what looks like the PID (not the username), so it should be unique. This condition occurs when the cache is fully populated and is close to the max size (20G in my case). This made me think that the race is between one users' process doing a cleanup, with another users' process doing a compile.

 I saw this error again today. The $CCACHE_DIR/f/stats file has the max size counter re-set to 65536 (i.e. 1024*1024/16). It should be 1310720 (i.e. 20*1024*1024/16). I am looking at the code a bit more, but do the experts know what conditions would cause the max size counter to get reset? Is it possible for the stats file to get deleted? It looks like the subroutine that updates the compile statistics creates a tmp file that it "renames" to the actual stats file (i.e. atomic rename under NFS). Is it possible that the tmp stats file could be getting cleaned up?

 Thanks.

----- Original Message -----
From: David Givone
Sent: 11/02/12 02:35 PM
To: Lalit Chhabra
Subject: Re: [ccache] ccache: FATAL: Could not create xxx.o.tmp.stdout.yyy.zzz (permission denied?)

 Verify that the tmp file is unique based on the machine and user. If it does not have the user and machine in the name it *may* be possible for a race condition.

 There was a race condition that I fixed that could occur when the cache directory did not exist and multiple processes were trying to create it for the first time. The fix was in a function that creates paths so it may have been possible that a race condition could occur when any directory was created. When the error occurred was the cache fully populated (all the directories created)?

 On Thu, Nov 1, 2012 at 4:40 PM, Lalit Chhabra < lchhabra at linuxmail.org > wrote:
 Hi,

 We have deployed ccache 3.1.7 (over NFS), with the cache being shared amongst about 30-40 people. The cache max size is configured to be 20G. We primarily compile c++ files with g++, each of which take less than a couple of minutes to compile. This has mostly been very sucessful, except that every once in a while, we get the above FATAL error from ccache. (I have seen this 2 times, in the past 4 months that we have been using ccache.)

 My initial guess was that there is some race condition between one users' ccache process doing a clean-up, while another users' compile is going on. But looking at the code, temp files have to be at least one hour old for them to get cleaned up.

 Another odd thing that I have seen (just once so far), is that the cache max size got reduced to 18.8G (which seems to be close to 20G*15/16). This caused a lot more of the "FATAL: Could not create..." messages to appear, until I reset the cache size back to 20G.

 I am not sure if both of these observations are related. Any ideas what could be going on?

 Thanks,
 Lalit Chhabra
 _______________________________________________
 ccache mailing list
 ccache at lists.samba.org 
https://lists.samba.org/mailman/listinfo/ccache