[distcc] Re: Incorrect time stamp of generated object file when using distcc with makepp

Chris van Engelen cvengelen at aimsys.nl
Mon Aug 30 07:35:47 GMT 2004

On Fri, Aug 27, 2004 at 02:40:05PM +0100, Jean Delvare wrote:

 >>>>OK, so makepp does not store the correct timestamp. How exactly is it distcc's

It is not at all distcc's fault, its just a problem of using distcc's memory mapped
file IO together with makepp using the mtime of a generated object file as signature.
So it's just an unhappy interworking effect between two features which on their own
are really nice, but put together cause a problem.

Here is what exactly happens when distcc is using memory mapped file IO, and is used
in a compilation rule in a makepp build:

   - distcc receives the compiled object from a server
   - when using memory mapped file IO, distcc calls a function mmap, and then
     considers the file as being written to file, and therefore finishes.
   - then makepp, noticing that the compilation command completed successfully,
     gets the mtime of the generated object file, and stores this as signature in
     a file in a hidden directory.
   - however, the memory mapped file IO mechanism is not yet finished writing
     the file actually to disk, this may take between 2-10 seconds!. Accordingly,
     the mtime of the file is also updated and after the file is completely written,
     it is 2-10 seconds further in time than the time recorded by makepp.

I tested this in the most simple way:

   - I wrote a one line perl script (testFileInfo) which prints the mtime of a file:

       use FileInfo;
       my $finfo = file_info( "$ARGV[ 0 ]" );
       print( "file ", $finfo->absolute_filename, " mtime: ", $finfo->file_mtime, "\n" );

   - I then used this script in a compilation rule in the following way:

       distcc gcc command
       print the mtime of the generated object file
       sleep for 5 seconds
       print the mtime of the generated object file again

     The exact commands in the rule are:

	$(DIAG) Compiling $(input) to $(output)
	$(CXX) -c $(absolute-filename $(input)) $(CPPFLAGS) $(CXXFLAGS) -o $(output)
	/home/cvengelen/tmp/makepp-linux/testFileInfo ${output}
	sleep 5
	/home/cvengelen/tmp/makepp-linux/testFileInfo ${output}

   - Then, when compiling files which result in an object file larger than 65 kB, I clearly see
     that in almost all cases the mtime after the sleep is modified from before the sleep.
     For example:

       debug: Compiling ../../../cih/src/cih_PortGroup.C to ../cih/src/cih_PortGroup.o
       /swdev/tools/bin/distcc /swdev/tools/cross/bin/powerpc-8xx-linux-g++ -c /home/cvengelen/projects/amu/sw/rel-1.0.1/tih/cih/src/cih_PortGroup.C -I/home/cvengelen/projects/amu/sw/rel-1.0.1
-D_REENTRANT -DPOSIX4 -D_GNU_SOURCE -DLINUX -DUNIX -fsigned-char -fno-builtin -Wall -Wstrict-prototypes -Wmissing-prototypes -W -Winline -Wredundant-decls -DMPC8270 -DMAINCTRL -fno-operator-names -O
-o ../cih/src/cih_PortGroup.o
       /home/cvengelen/tmp/makepp-linux/testFileInfo ../cih/src/cih_PortGroup.o
       file /home/cvengelen/projects/amu/sw/rel-1.0.1/tih/compiler/gcc-linuxppc+mainctrl/cih/src/cih_PortGroup.o mtime: 1093848988
       sleep 5
       /home/cvengelen/tmp/makepp-linux/testFileInfo ../cih/src/cih_PortGroup.o
       file /home/cvengelen/projects/amu/sw/rel-1.0.1/tih/compiler/gcc-linuxppc+mainctrl/cih/src/cih_PortGroup.o mtime: 1093848991

      This happens for all files fo which the object file is larger than 65 kB

- I then switch off the memory mapped file IO mechanism of distcc by setting environment
    variable DISTCC_MMAP to 0, and then the mtime does no longer change:

       export DISTCC_MMAP=0

       execute makepp command with output:

       debug: Compiling ../../../cih/src/cih_PortGroup.C to ../cih/src/cih_PortGroup.o
       /swdev/tools/bin/distcc /swdev/tools/cross/bin/powerpc-8xx-linux-g++ -c /home/cvengelen/projects/amu/sw/rel-1.0.1/tih/cih/src/cih_PortGroup.C -I/home/cvengelen/projects/amu/sw/rel-1.0.1
-D_REENTRANT -DPOSIX4 -D_GNU_SOURCE -DLINUX -DUNIX -fsigned-char -fno-builtin -Wall -Wstrict-prototypes -Wmissing-prototypes -W -Winline -Wredundant-decls -DMPC8270 -DMAINCTRL -fno-operator-names -O
-o ../cih/src/cih_PortGroup.o
       /home/cvengelen/tmp/makepp-linux/testFileInfo ../cih/src/cih_PortGroup.o
       file /home/cvengelen/projects/amu/sw/rel-1.0.1/tih/compiler/gcc-linuxppc+mainctrl/cih/src/cih_PortGroup.o mtime: 1093849187
       sleep 5
       /home/cvengelen/tmp/makepp-linux/testFileInfo ../cih/src/cih_PortGroup.o
       file /home/cvengelen/projects/amu/sw/rel-1.0.1/tih/compiler/gcc-linuxppc+mainctrl/cih/src/cih_PortGroup.o mtime: 1093849187

     The mtime of ALL generated object files no longer is changed after the sleep 5
     when using DISTCC_MMAP=0. This is 100% reproducable.

So this proves beyond any doubt that it is the memory mapped file IO used by distcc
which causes makepp to store a mtime signature, which is not correct for object files
larger than 65 kB, causing recompilation the next time around.

So this is just a fine example of how two fancy mechanisms may lead to unexpected
interworking problems, in this case between distcc using memory mapped file IO and
makepp using the mtime of a generated file.

Also not that this problem is not related at all to any time differences between
distcc servers and hosts: this may have caused other problems, but not this one!

For your information, the related bit of distcc code can be found in distcc source
file pump.c, function dcc_r_bulk_plain, which I have attached to this mail. Also
notice the check on file size 65536, which explains why I only notice this problem
for object files with size larger than 65 kB.

Hope this helps explaining the problem. Please let me know if you need
any further information.
Chris van Engelen
AimSys bv
Hilversum, The Netherlands
Phone: +31 35 689 1935
mailto:cvengelen at aimsys.nl
-------------- next part --------------
/* -*- c-file-style: "java"; indent-tabs-mode: nil; fill-column: 78 -*-
 * distcc -- A simple distributed compiler system
 * Copyright (C) 2003 by Martin Pool <mbp at samba.org>
 * This program is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public License as
 * published by the Free Software Foundation; either version 2 of the
 * License, or (at your option) any later version.
 * This program is distributed in the hope that it will be useful, but
 * WITHOUT ANY WARRANTY; without even the implied warranty of
 * General Public License for more details.
 * You should have received a copy of the GNU General Public License
 * along with this program; if not, write to the Free Software
 * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307
 * USA

#include "config.h"

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <fcntl.h>
#include <errno.h>

#include <sys/stat.h>
#include <sys/types.h>
#include <sys/socket.h>
#  include <sys/select.h>
#include <sys/time.h>
#  include <sys/mman.h>

#include <netinet/in.h>
#include <netinet/tcp.h>

#include "distcc.h"
#include "trace.h"
#include "io.h"
#include "util.h"
#include "exitcode.h"

 * @file
 * Transfer of bulk data (source, object code)

 * Receive data from the network to the filesystem.  Uses mmap if the data is
 * large and we can, otherwise reads into a mallocd buffer and writes it out.
 * @p out_fd should be open read/write.
static int dcc_r_bulk_plain(int out_fd,
                            int in_fd,
                            unsigned len)
    int ret;
    char *buf = NULL;
    int is_mmapped = 0;

    if (len == 0)
        return 0;               /* just check */

    if (len >= 65536  &&  dcc_want_mmap()) {
#ifdef HAVE_MMAP
        /* First truncate the file, so that we have space to write to it.  This
         * will fail if we're e.g. writing to stdout. */
        buf = mmap(NULL,            /* suggested address */
                   len, PROT_WRITE|PROT_READ,
                   MAP_SHARED, out_fd,
                   0);              /* offset */
        if (buf == MAP_FAILED) {
            rs_trace("mmap output fd%d failed: %s", out_fd, strerror(errno));
        } else {
            is_mmapped = 1;

            rs_trace("receive %d bytes using mmap", len);

            /* Now make it big enough to write to.  You fall through the floor
             * if you try to write to nonexistent parts of the file.
             * Because this routine is for uncompressed data we know ahead of
             * time exactly how much space we will need.
            if (ftruncate(out_fd, len) == -1) {
                rs_log_error("ftruncate fd%d failed: %s", out_fd,
                ret = EXIT_IO_ERROR;
                goto out;

    if (!is_mmapped) {
        if ((buf = malloc(len)) == NULL) {
            rs_log_error("failed to allocate receipt buffer");
            ret = EXIT_OUT_OF_MEMORY;
            goto out;

    if ((ret = dcc_readx(in_fd, buf, len)) != 0)
        goto out;

    if (!is_mmapped) {
        ret = dcc_writex(out_fd, buf, len);

    if (is_mmapped) {
        if (munmap(buf, len) == -1) {
            rs_log_error("munmap (output) failed: %s",
            ret = ret ? ret : EXIT_IO_ERROR;
    } else { 
    return ret;

 * Common routine for reading a file body used by both fifos and
 * regular files.
 * The output file descriptor is not closed, because this routine is
 * also used for copying onto stderr.
int dcc_r_bulk(int ofd,
               int ifd,
               unsigned f_size,
               enum dcc_compress compression)
    if (f_size == 0)
        return 0;               /* don't decompress nothing */
    if (compression == DCC_COMPRESS_NONE) {
        return dcc_r_bulk_plain(ofd, ifd, f_size);
    } else if (compression == DCC_COMPRESS_LZO1X) {
        return dcc_r_bulk_lzo1x(ofd, ifd, f_size);
    } else {
        rs_fatal("impossible compression %d", compression);

 * Copy @p n bytes from @p ifd to @p ofd.
 * Does not use sendfile(), so either one may be a socket.
 * In the current code at least one of the files will always be a regular
 * (disk) file, even though it may not be mmapable.  That should mean that
 * writes to it will always complete immediately.  That in turn means that on
 * each pass through the main loop we ought to either completely fill our
 * buffer, or completely drain it, depending on which one is the disk.
 * In future we may put back the ability to feed the compiler from a fifo, in
 * which case it may be that the writes don't complete.
 * We might try selecting on both buffers and handling whichever is ready.
 * This would require some approximation to a circular buffer though, which
 * might be more complex.
int dcc_pump_readwrite(int ofd, int ifd, size_t n)
    static char buf[262144];    /* we're not recursive */
    char *p;
    ssize_t r_in, r_out, wanted;
    int ret;

    while (n > 0) {
        wanted = (n > sizeof buf) ? (sizeof buf) : n;
        r_in = read(ifd, buf, (size_t) wanted);

        if (r_in == -1 && errno == EAGAIN) {
            if ((ret = dcc_select_for_read(ifd, dcc_io_timeout)) != 0)
                return ret;
        } else if (r_in == -1 && errno == EINTR) {
        } else if (r_in == -1) {
            rs_log_error("failed to read %ld bytes: %s",
                         (long) wanted, strerror(errno));
            return EXIT_IO_ERROR;
        } else if (r_in == 0) {
            rs_log_error("unexpected eof on fd%d", ifd);
            return EXIT_IO_ERROR;
        n -= r_in;
        p = buf;

        /* We now have r_in bytes waiting to go out, starting at p.  Keep
         * going until they're all written out. */

        while (r_in > 0) {
            r_out = write(ofd, p, (size_t) r_in);
            if (r_out == -1 && errno == EAGAIN) {
                if ((ret = dcc_select_for_write(ofd, dcc_io_timeout)) != 0)
                    return ret;
            } else if (r_out == -1 && errno == EINTR) {
            } else if (r_out == -1  ||  r_out == 0) {
                rs_log_error("failed to write: %s", strerror(errno));
                return EXIT_IO_ERROR;
            r_in -= r_out;
            p += r_out;

    return 0;

More information about the distcc mailing list