[distcc] Repeatable .o and .so checksums with distcc

[Jeff Kilpatrick]

Hello,

At my work, we've just begun to investigate how much of an impact that
distcc will have on our builds.

We typically perform 200 builds a week, ranging from a thousand lines of
code, up to 600,000 lines of code each. Our back end build scripts are based
on python, and use Linux make to build. We are running VMWare images on a
blade cluster, and each of our three new build servers have 20Ghz processing
power, with 4G of RAM. Our primary build environments are loop back ISOs,
from a central CIFS server, and are unioned together with unionfs. Our
source code is then copied into this environment, and we proceed with our
build, using chroot to enter our build environment. Our 'distcc' machines
use the same loop back system, with only our OS and distcc being accessible.


One of the most important things for our builds, due to the market that we
are in, is that our builds must be reproducible, with repeatable md5sums on
our shared objects, based on the same label and same dependencies. In our
recent tests, we were able to take a particular build from 24 minutes to 14
minutes, then finally 5 minutes, using distcc and adjusting our VMs.
However, when performing an md5sum on our final shared objects / object
files, the checksums change every build. We dropped down to just using g++
to perform our linking, all locally, but our object files are still
mismatching.

In the object files' `objdump -s` output, it appears that an entry is being
made into all our object files with the following syntax "distccd_XXXXX",
with XXXXX being a seemingly random combination of characters.

In the same object file, compiled locally without distcc, we get a rather
generic <built-in> placeholder.

I've reviewed the source code for distcc, and seen a few references to this
distccd_xxxxx. Unfortunately, I'm not a programmer, and thus am at a loss on
how to further troubleshoot this, or even if its possible to get consistent
checksums with distcc.


Versions
=======
g++ (Gentoo 4.3.2-r4 p1.8, pie-10.1.5) 4.3.2

distcc 3.1 i686-pc-linux-gnu
  (protocols 1, 2 and 3) (default port 3632)
  built Mar 29 2010 10:55:35

Kernel: 2.6.9-89.ELsmp

Command being issued:
      DISTCC_VERBOSE=1 make -j24 CXX="distcc"

Here's the partial output of objdump -s:
 04f0 00030000 5f6d6f76 655f636f 6e737472  ...._move_constr
 0500 7563745f 66776b2e 68000300 00474454  uct_fwk.h....GDT
 0510 79706573 2e68000a 00007365 72646566  ypes.h....serdef
 0520 732e6800 01000073 75666669 782e6870  s.h....suffix.hp
 0530 70000b00 00646973 74636364 5f616333  p....distccd_ac3
 0540 31633936 612e6969 000c0000 61646c5f  1c96a.ii....adl_
 0550 62617272 6965722e 68707000 0d000062  barrier.hpp....b
 0560 6f6f6c5f 6677642e 68707000 0e000069  ool_fwd.hpp....i
 0570 6e746567 72616c5f 635f7461 672e6870  ntegral_c_tag.hp
 0580 70000e00 00766f69 645f6677 642e6870  p....void_fwd.hp

Thank you for reviewing my issue.

-Jeff
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/distcc/attachments/20100629/46b594d3/attachment.html>