[distcc] Re: Incorrect time stamp of generated object file when
using distcc with makepp
Chris van Engelen
cvengelen at aimsys.nl
Mon Aug 30 07:35:47 GMT 2004
On Fri, Aug 27, 2004 at 02:40:05PM +0100, Jean Delvare wrote:
>>>>OK, so makepp does not store the correct timestamp. How exactly is it distcc's
>>>>fault?
It is not at all distcc's fault, its just a problem of using distcc's memory mapped
file IO together with makepp using the mtime of a generated object file as signature.
So it's just an unhappy interworking effect between two features which on their own
are really nice, but put together cause a problem.
Here is what exactly happens when distcc is using memory mapped file IO, and is used
in a compilation rule in a makepp build:
- distcc receives the compiled object from a server
- when using memory mapped file IO, distcc calls a function mmap, and then
considers the file as being written to file, and therefore finishes.
- then makepp, noticing that the compilation command completed successfully,
gets the mtime of the generated object file, and stores this as signature in
a file in a hidden directory.
- however, the memory mapped file IO mechanism is not yet finished writing
the file actually to disk, this may take between 2-10 seconds!. Accordingly,
the mtime of the file is also updated and after the file is completely written,
it is 2-10 seconds further in time than the time recorded by makepp.
I tested this in the most simple way:
- I wrote a one line perl script (testFileInfo) which prints the mtime of a file:
use FileInfo;
my $finfo = file_info( "$ARGV[ 0 ]" );
print( "file ", $finfo->absolute_filename, " mtime: ", $finfo->file_mtime, "\n" );
- I then used this script in a compilation rule in the following way:
distcc gcc command
print the mtime of the generated object file
sleep for 5 seconds
print the mtime of the generated object file again
The exact commands in the rule are:
$(DIAG) Compiling $(input) to $(output)
$(CXX) -c $(absolute-filename $(input)) $(CPPFLAGS) $(CXXFLAGS) -o $(output)
/home/cvengelen/tmp/makepp-linux/testFileInfo ${output}
sleep 5
/home/cvengelen/tmp/makepp-linux/testFileInfo ${output}
- Then, when compiling files which result in an object file larger than 65 kB, I clearly see
that in almost all cases the mtime after the sleep is modified from before the sleep.
For example:
debug: Compiling ../../../cih/src/cih_PortGroup.C to ../cih/src/cih_PortGroup.o
/swdev/tools/bin/distcc /swdev/tools/cross/bin/powerpc-8xx-linux-g++ -c /home/cvengelen/projects/amu/sw/rel-1.0.1/tih/cih/src/cih_PortGroup.C -I/home/cvengelen/projects/amu/sw/rel-1.0.1
-D_REENTRANT -DPOSIX4 -D_GNU_SOURCE -DLINUX -DUNIX -fsigned-char -fno-builtin -Wall -Wstrict-prototypes -Wmissing-prototypes -W -Winline -Wredundant-decls -DMPC8270 -DMAINCTRL -fno-operator-names -O
-o ../cih/src/cih_PortGroup.o
/home/cvengelen/tmp/makepp-linux/testFileInfo ../cih/src/cih_PortGroup.o
file /home/cvengelen/projects/amu/sw/rel-1.0.1/tih/compiler/gcc-linuxppc+mainctrl/cih/src/cih_PortGroup.o mtime: 1093848988
sleep 5
/home/cvengelen/tmp/makepp-linux/testFileInfo ../cih/src/cih_PortGroup.o
file /home/cvengelen/projects/amu/sw/rel-1.0.1/tih/compiler/gcc-linuxppc+mainctrl/cih/src/cih_PortGroup.o mtime: 1093848991
This happens for all files fo which the object file is larger than 65 kB
- I then switch off the memory mapped file IO mechanism of distcc by setting environment
variable DISTCC_MMAP to 0, and then the mtime does no longer change:
export DISTCC_MMAP=0
execute makepp command with output:
debug: Compiling ../../../cih/src/cih_PortGroup.C to ../cih/src/cih_PortGroup.o
/swdev/tools/bin/distcc /swdev/tools/cross/bin/powerpc-8xx-linux-g++ -c /home/cvengelen/projects/amu/sw/rel-1.0.1/tih/cih/src/cih_PortGroup.C -I/home/cvengelen/projects/amu/sw/rel-1.0.1
-D_REENTRANT -DPOSIX4 -D_GNU_SOURCE -DLINUX -DUNIX -fsigned-char -fno-builtin -Wall -Wstrict-prototypes -Wmissing-prototypes -W -Winline -Wredundant-decls -DMPC8270 -DMAINCTRL -fno-operator-names -O
-o ../cih/src/cih_PortGroup.o
/home/cvengelen/tmp/makepp-linux/testFileInfo ../cih/src/cih_PortGroup.o
file /home/cvengelen/projects/amu/sw/rel-1.0.1/tih/compiler/gcc-linuxppc+mainctrl/cih/src/cih_PortGroup.o mtime: 1093849187
sleep 5
/home/cvengelen/tmp/makepp-linux/testFileInfo ../cih/src/cih_PortGroup.o
file /home/cvengelen/projects/amu/sw/rel-1.0.1/tih/compiler/gcc-linuxppc+mainctrl/cih/src/cih_PortGroup.o mtime: 1093849187
The mtime of ALL generated object files no longer is changed after the sleep 5
when using DISTCC_MMAP=0. This is 100% reproducable.
So this proves beyond any doubt that it is the memory mapped file IO used by distcc
which causes makepp to store a mtime signature, which is not correct for object files
larger than 65 kB, causing recompilation the next time around.
So this is just a fine example of how two fancy mechanisms may lead to unexpected
interworking problems, in this case between distcc using memory mapped file IO and
makepp using the mtime of a generated file.
Also not that this problem is not related at all to any time differences between
distcc servers and hosts: this may have caused other problems, but not this one!
For your information, the related bit of distcc code can be found in distcc source
file pump.c, function dcc_r_bulk_plain, which I have attached to this mail. Also
notice the check on file size 65536, which explains why I only notice this problem
for object files with size larger than 65 kB.
Hope this helps explaining the problem. Please let me know if you need
any further information.
--
Chris van Engelen
AimSys bv
Hilversum, The Netherlands
Phone: +31 35 689 1935
mailto:cvengelen at aimsys.nl
-------------- next part --------------
/* -*- c-file-style: "java"; indent-tabs-mode: nil; fill-column: 78 -*-
*
* distcc -- A simple distributed compiler system
*
* Copyright (C) 2003 by Martin Pool <mbp at samba.org>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License as
* published by the Free Software Foundation; either version 2 of the
* License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful, but
* WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307
* USA
*/
#include "config.h"
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <fcntl.h>
#include <errno.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/socket.h>
#ifdef HAVE_SYS_SELECT_H
# include <sys/select.h>
#endif
#include <sys/time.h>
#ifdef HAVE_SYS_MMAN_H
# include <sys/mman.h>
#endif
#include <netinet/in.h>
#include <netinet/tcp.h>
#include "distcc.h"
#include "trace.h"
#include "io.h"
#include "util.h"
#include "exitcode.h"
/**
* @file
* Transfer of bulk data (source, object code)
**/
/**
* Receive data from the network to the filesystem. Uses mmap if the data is
* large and we can, otherwise reads into a mallocd buffer and writes it out.
*
* @p out_fd should be open read/write.
**/
static int dcc_r_bulk_plain(int out_fd,
int in_fd,
unsigned len)
{
int ret;
char *buf = NULL;
int is_mmapped = 0;
if (len == 0)
return 0; /* just check */
if (len >= 65536 && dcc_want_mmap()) {
#ifdef HAVE_MMAP
/* First truncate the file, so that we have space to write to it. This
* will fail if we're e.g. writing to stdout. */
buf = mmap(NULL, /* suggested address */
len, PROT_WRITE|PROT_READ,
MAP_SHARED, out_fd,
0); /* offset */
if (buf == MAP_FAILED) {
rs_trace("mmap output fd%d failed: %s", out_fd, strerror(errno));
} else {
is_mmapped = 1;
rs_trace("receive %d bytes using mmap", len);
/* Now make it big enough to write to. You fall through the floor
* if you try to write to nonexistent parts of the file.
*
* Because this routine is for uncompressed data we know ahead of
* time exactly how much space we will need.
*/
if (ftruncate(out_fd, len) == -1) {
rs_log_error("ftruncate fd%d failed: %s", out_fd,
strerror(errno));
ret = EXIT_IO_ERROR;
goto out;
}
}
#endif
}
if (!is_mmapped) {
if ((buf = malloc(len)) == NULL) {
rs_log_error("failed to allocate receipt buffer");
ret = EXIT_OUT_OF_MEMORY;
goto out;
}
}
if ((ret = dcc_readx(in_fd, buf, len)) != 0)
goto out;
if (!is_mmapped) {
ret = dcc_writex(out_fd, buf, len);
}
out:
if (is_mmapped) {
if (munmap(buf, len) == -1) {
rs_log_error("munmap (output) failed: %s",
strerror(errno));
ret = ret ? ret : EXIT_IO_ERROR;
}
} else {
free(buf);
}
return ret;
}
/**
* Common routine for reading a file body used by both fifos and
* regular files.
*
* The output file descriptor is not closed, because this routine is
* also used for copying onto stderr.
**/
int dcc_r_bulk(int ofd,
int ifd,
unsigned f_size,
enum dcc_compress compression)
{
if (f_size == 0)
return 0; /* don't decompress nothing */
if (compression == DCC_COMPRESS_NONE) {
return dcc_r_bulk_plain(ofd, ifd, f_size);
} else if (compression == DCC_COMPRESS_LZO1X) {
return dcc_r_bulk_lzo1x(ofd, ifd, f_size);
} else {
rs_fatal("impossible compression %d", compression);
}
}
/**
* Copy @p n bytes from @p ifd to @p ofd.
*
* Does not use sendfile(), so either one may be a socket.
*
* In the current code at least one of the files will always be a regular
* (disk) file, even though it may not be mmapable. That should mean that
* writes to it will always complete immediately. That in turn means that on
* each pass through the main loop we ought to either completely fill our
* buffer, or completely drain it, depending on which one is the disk.
*
* In future we may put back the ability to feed the compiler from a fifo, in
* which case it may be that the writes don't complete.
*
* We might try selecting on both buffers and handling whichever is ready.
* This would require some approximation to a circular buffer though, which
* might be more complex.
**/
int dcc_pump_readwrite(int ofd, int ifd, size_t n)
{
static char buf[262144]; /* we're not recursive */
char *p;
ssize_t r_in, r_out, wanted;
int ret;
while (n > 0) {
wanted = (n > sizeof buf) ? (sizeof buf) : n;
r_in = read(ifd, buf, (size_t) wanted);
if (r_in == -1 && errno == EAGAIN) {
if ((ret = dcc_select_for_read(ifd, dcc_io_timeout)) != 0)
return ret;
else
continue;
} else if (r_in == -1 && errno == EINTR) {
continue;
} else if (r_in == -1) {
rs_log_error("failed to read %ld bytes: %s",
(long) wanted, strerror(errno));
return EXIT_IO_ERROR;
} else if (r_in == 0) {
rs_log_error("unexpected eof on fd%d", ifd);
return EXIT_IO_ERROR;
}
n -= r_in;
p = buf;
/* We now have r_in bytes waiting to go out, starting at p. Keep
* going until they're all written out. */
while (r_in > 0) {
r_out = write(ofd, p, (size_t) r_in);
if (r_out == -1 && errno == EAGAIN) {
if ((ret = dcc_select_for_write(ofd, dcc_io_timeout)) != 0)
return ret;
else
continue;
} else if (r_out == -1 && errno == EINTR) {
continue;
} else if (r_out == -1 || r_out == 0) {
rs_log_error("failed to write: %s", strerror(errno));
return EXIT_IO_ERROR;
}
r_in -= r_out;
p += r_out;
}
}
return 0;
}
More information about the distcc
mailing list