[distcc] Fwd: Repeatable .o and .so checksums with distcc

Tue Jun 29 07:52:10 MDT 2010

Yes, I have tried both pump and regular mode, and both behave the same way.

A lot of the projects that I will be compiling include boost, and I believe
that the pump fails on those, and falls back to regular mode.

-Jeff

On Tue, Jun 29, 2010 at 10:48 AM, Fergus Henderson <fergus at google.com>wrote:

> Did you try using pump mode?
> That should give you a better build speed-up and may also avoid this issue.
>
> On Jun 29, 2010 6:32 AM, "Jeff Kilpatrick" <kilpatrick.jeff at gmail.com>
> wrote:
> > Oops, my original response went directly to Ihar, rather than to the
> list.
> >
> > ----
> >
> >
> >
> > Thank you for your response.
> >
> > We do have a tool internally that could 'scrub' the object file of its
> > dynamic symbols, and could be adapted for this purpose. However, I'm
> > hesitant to modify anything with the .o and .so with an external tool, as
> in
> > some cases, it may be hiding a legitimate issue. Once an exception makes
> it
> > into the code, its tempting to continue adding exceptions to fix issues.
> > Before you know it, you have 600 branches with unique 'fixes' to them :)
> >
> > Once we get a consistent checksum on the .o and .so files, they'll be
> > packaged into a .iso, which will also need to be repeatable. This can be
> > challenging as well, since attributes on the files can affect the final
> > checksum.
> >
> > -Jeff
> >
> >
> > On Tue, Jun 29, 2010 at 6:58 AM, Ihar `Philips` Filipau <
> > thephilips at gmail.com> wrote:
> >
> >> Hi Jeff!
> >>
> >> You can try to collect the check-sum only for the ELF segments which are
> >> actually derived from the the source code, omitting the segments with
> the
> >> extra compiler's info. I do not know any ready tool for the purpose, but
> >> coding something like this - print on stdout all segments except the
> >> black-listed - shouldn't be too complicated.
> >>
> >>
> >> On Tue, Jun 29, 2010 at 11:41 AM, Jeff Kilpatrick <
> >> kilpatrick.jeff at gmail.com> wrote:
> >>
> >>> Thank you for your response.
> >>>
> >>> Yes, this is the only difference in the object file. We've taken great
> >>> pains over the last few years, removing anything that would cause
> checksums
> >>> to mismatch.
> >>>
> >>> I will do some research myself, and talk to a few developers to see if
> >>> they can help me.
> >>>
> >>> Thanks
> >>> -Jeff
> >>>
> >>>
> >>> On Tue, Jun 29, 2010 at 1:32 AM, Martin Pool <mbp at sourcefrog.net>
> wrote:
> >>>
> >>>> On 29 June 2010 13:02, Jeff Kilpatrick <kilpatrick.jeff at gmail.com>
> >>>> wrote:
> >>>> > Hello,
> >>>> >
> >>>> > At my work, we've just begun to investigate how much of an impact
> that
> >>>> > distcc will have on our builds.
> >>>> >
> >>>> > We typically perform 200 builds a week, ranging from a thousand
> lines
> >>>> of
> >>>> > code, up to 600,000 lines of code each. Our back end build scripts
> are
> >>>> based
> >>>> > on python, and use Linux make to build. We are running VMWare images
> on
> >>>> a
> >>>> > blade cluster, and each of our three new build servers have 20Ghz
> >>>> processing
> >>>> > power, with 4G of RAM. Our primary build environments are loop back
> >>>> ISOs,
> >>>> > from a central CIFS server, and are unioned together with unionfs.
> Our
> >>>> > source code is then copied into this environment, and we proceed
> with
> >>>> our
> >>>> > build, using chroot to enter our build environment. Our 'distcc'
> >>>> machines
> >>>> > use the same loop back system, with only our OS and distcc being
> >>>> accessible.
> >>>>
> >>>> That's pretty cool.
> >>>>
> >>>> > One of the most important things for our builds, due to the market
> that
> >>>> we
> >>>> > are in, is that our builds must be reproducible, with repeatable
> >>>> md5sums on
> >>>> > our shared objects, based on the same label and same dependencies.
> In
> >>>> our
> >>>> > recent tests, we were able to take a particular build from 24
> minutes
> >>>> to 14
> >>>> > minutes, then finally 5 minutes, using distcc and adjusting our VMs.
> >>>> > However, when performing an md5sum on our final shared objects /
> object
> >>>> > files, the checksums change every build. We dropped down to just
> using
> >>>> g++
> >>>> > to perform our linking, all locally, but our object files are still
> >>>> > mismatching.
> >>>> >
> >>>> > In the object files' `objdump -s` output, it appears that an entry
> is
> >>>> being
> >>>> > made into all our object files with the following syntax
> >>>> "distccd_XXXXX",
> >>>> > with XXXXX being a seemingly random combination of characters.
> >>>>
> >>>> Hi Jeff,
> >>>>
> >>>> I think this is coming from gcc recording the input file name in the
> >>>> object file. distccd_xxxx.ii is the temporary file name used on the
> >>>> server.
> >>>>
> >>>> > In the same object file, compiled locally without distcc, we get a
> >>>> rather
> >>>> > generic <built-in> placeholder.
> >>>>
> >>>> I think this means it's coming from the builtin preprocessor.
> >>>>
> >>>> I probably won't have time to work on this myself but if you have a
> >>>> programmer interested in it there are two possible avenues:
> >>>>
> >>>> - make gcc read from a file called <built-in> in a temporary
> subdirectory
> >>>>
> >>>> - find some way to stop it recording the compiler input file name
> >>>>
> >>>> Is that the only difference in the object files? It's pretty common
> >>>> for compilers to also record something about the time the compilation
> >>>> was run or for source files to build this in, which would mean they
> >>>> change every time.
> >>>>
> >>>> >
> >>>> > I've reviewed the source code for distcc, and seen a few references
> to
> >>>> this
> >>>> > distccd_xxxxx. Unfortunately, I'm not a programmer, and thus am at a
> >>>> loss on
> >>>> > how to further troubleshoot this, or even if its possible to get
> >>>> consistent
> >>>> > checksums with distcc.
> >>>> >
> >>>> >
> >>>> > Versions
> >>>> > =======
> >>>> > g++ (Gentoo 4.3.2-r4 p1.8, pie-10.1.5) 4.3.2
> >>>> >
> >>>> > distcc 3.1 i686-pc-linux-gnu
> >>>> > (protocols 1, 2 and 3) (default port 3632)
> >>>> > built Mar 29 2010 10:55:35
> >>>> >
> >>>> > Kernel: 2.6.9-89.ELsmp
> >>>> >
> >>>> > Command being issued:
> >>>> > DISTCC_VERBOSE=1 make -j24 CXX="distcc"
> >>>> >
> >>>> > Here's the partial output of objdump -s:
> >>>> > 04f0 00030000 5f6d6f76 655f636f 6e737472 ...._move_constr
> >>>> > 0500 7563745f 66776b2e 68000300 00474454 uct_fwk.h....GDT
> >>>> > 0510 79706573 2e68000a 00007365 72646566 ypes.h....serdef
> >>>> > 0520 732e6800 01000073 75666669 782e6870 s.h....suffix.hp
> >>>> > 0530 70000b00 00646973 74636364 5f616333 p....distccd_ac3
> >>>> > 0540 31633936 612e6969 000c0000 61646c5f 1c96a.ii....adl_
> >>>> > 0550 62617272 6965722e 68707000 0d000062 barrier.hpp....b
> >>>> > 0560 6f6f6c5f 6677642e 68707000 0e000069 ool_fwd.hpp....i
> >>>> > 0570 6e746567 72616c5f 635f7461 672e6870 ntegral_c_tag.hp
> >>>> > 0580 70000e00 00766f69 645f6677 642e6870 p....void_fwd.hp
> >>>> >
> >>>> > Thank you for reviewing my issue.
> >>>> >
> >>>> > -Jeff
> >>>> >
> >>>> > __
> >>>> > distcc mailing list http://distcc.samba.org/
> >>>> > To unsubscribe or change options:
> >>>> > https://lists.samba.org/mailman/listinfo/distcc
> >>>> >
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Martin
> >>>>
> >>>
> >>>
> >>> __
> >>> distcc mailing list http://distcc.samba.org/
> >>> To unsubscribe or change options:
> >>> https://lists.samba.org/mailman/listinfo/distcc
> >>>
> >>
> >>
> >>
> >> --
> >> Don't walk behind me, I may not lead.
> >> Don't walk in front of me, I may not follow.
> >> Just walk beside me and be my friend.
> >> -- Albert Camus (attributed to)
> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/distcc/attachments/20100629/e6731707/attachment.html>