[distcc] Fwd: Repeatable .o and .so checksums with distcc

Tue Jun 29 07:48:54 MDT 2010

Did you try using pump mode?
That should give you a better build speed-up and may also avoid this issue.

On Jun 29, 2010 6:32 AM, "Jeff Kilpatrick" <kilpatrick.jeff at gmail.com>
wrote:
> Oops, my original response went directly to Ihar, rather than to the list.
>
> ----
>
>
>
> Thank you for your response.
>
> We do have a tool internally that could 'scrub' the object file of its
> dynamic symbols, and could be adapted for this purpose. However, I'm
> hesitant to modify anything with the .o and .so with an external tool, as
in
> some cases, it may be hiding a legitimate issue. Once an exception makes
it
> into the code, its tempting to continue adding exceptions to fix issues.
> Before you know it, you have 600 branches with unique 'fixes' to them :)
>
> Once we get a consistent checksum on the .o and .so files, they'll be
> packaged into a .iso, which will also need to be repeatable. This can be
> challenging as well, since attributes on the files can affect the final
> checksum.
>
> -Jeff
>
>
> On Tue, Jun 29, 2010 at 6:58 AM, Ihar `Philips` Filipau <
> thephilips at gmail.com> wrote:
>
>> Hi Jeff!
>>
>> You can try to collect the check-sum only for the ELF segments which are
>> actually derived from the the source code, omitting the segments with the
>> extra compiler's info. I do not know any ready tool for the purpose, but
>> coding something like this - print on stdout all segments except the
>> black-listed - shouldn't be too complicated.
>>
>>
>> On Tue, Jun 29, 2010 at 11:41 AM, Jeff Kilpatrick <
>> kilpatrick.jeff at gmail.com> wrote:
>>
>>> Thank you for your response.
>>>
>>> Yes, this is the only difference in the object file. We've taken great
>>> pains over the last few years, removing anything that would cause
checksums
>>> to mismatch.
>>>
>>> I will do some research myself, and talk to a few developers to see if
>>> they can help me.
>>>
>>> Thanks
>>> -Jeff
>>>
>>>
>>> On Tue, Jun 29, 2010 at 1:32 AM, Martin Pool <mbp at sourcefrog.net> wrote:
>>>
>>>> On 29 June 2010 13:02, Jeff Kilpatrick <kilpatrick.jeff at gmail.com>
>>>> wrote:
>>>> > Hello,
>>>> >
>>>> > At my work, we've just begun to investigate how much of an impact
that
>>>> > distcc will have on our builds.
>>>> >
>>>> > We typically perform 200 builds a week, ranging from a thousand lines
>>>> of
>>>> > code, up to 600,000 lines of code each. Our back end build scripts
are
>>>> based
>>>> > on python, and use Linux make to build. We are running VMWare images
on
>>>> a
>>>> > blade cluster, and each of our three new build servers have 20Ghz
>>>> processing
>>>> > power, with 4G of RAM. Our primary build environments are loop back
>>>> ISOs,
>>>> > from a central CIFS server, and are unioned together with unionfs.
Our
>>>> > source code is then copied into this environment, and we proceed with
>>>> our
>>>> > build, using chroot to enter our build environment. Our 'distcc'
>>>> machines
>>>> > use the same loop back system, with only our OS and distcc being
>>>> accessible.
>>>>
>>>> That's pretty cool.
>>>>
>>>> > One of the most important things for our builds, due to the market
that
>>>> we
>>>> > are in, is that our builds must be reproducible, with repeatable
>>>> md5sums on
>>>> > our shared objects, based on the same label and same dependencies. In
>>>> our
>>>> > recent tests, we were able to take a particular build from 24 minutes
>>>> to 14
>>>> > minutes, then finally 5 minutes, using distcc and adjusting our VMs.
>>>> > However, when performing an md5sum on our final shared objects /
object
>>>> > files, the checksums change every build. We dropped down to just
using
>>>> g++
>>>> > to perform our linking, all locally, but our object files are still
>>>> > mismatching.
>>>> >
>>>> > In the object files' `objdump -s` output, it appears that an entry is
>>>> being
>>>> > made into all our object files with the following syntax
>>>> "distccd_XXXXX",
>>>> > with XXXXX being a seemingly random combination of characters.
>>>>
>>>> Hi Jeff,
>>>>
>>>> I think this is coming from gcc recording the input file name in the
>>>> object file. distccd_xxxx.ii is the temporary file name used on the
>>>> server.
>>>>
>>>> > In the same object file, compiled locally without distcc, we get a
>>>> rather
>>>> > generic <built-in> placeholder.
>>>>
>>>> I think this means it's coming from the builtin preprocessor.
>>>>
>>>> I probably won't have time to work on this myself but if you have a
>>>> programmer interested in it there are two possible avenues:
>>>>
>>>> - make gcc read from a file called <built-in> in a temporary
subdirectory
>>>>
>>>> - find some way to stop it recording the compiler input file name
>>>>
>>>> Is that the only difference in the object files? It's pretty common
>>>> for compilers to also record something about the time the compilation
>>>> was run or for source files to build this in, which would mean they
>>>> change every time.
>>>>
>>>> >
>>>> > I've reviewed the source code for distcc, and seen a few references
to
>>>> this
>>>> > distccd_xxxxx. Unfortunately, I'm not a programmer, and thus am at a
>>>> loss on
>>>> > how to further troubleshoot this, or even if its possible to get
>>>> consistent
>>>> > checksums with distcc.
>>>> >
>>>> >
>>>> > Versions
>>>> > =======
>>>> > g++ (Gentoo 4.3.2-r4 p1.8, pie-10.1.5) 4.3.2
>>>> >
>>>> > distcc 3.1 i686-pc-linux-gnu
>>>> > (protocols 1, 2 and 3) (default port 3632)
>>>> > built Mar 29 2010 10:55:35
>>>> >
>>>> > Kernel: 2.6.9-89.ELsmp
>>>> >
>>>> > Command being issued:
>>>> > DISTCC_VERBOSE=1 make -j24 CXX="distcc"
>>>> >
>>>> > Here's the partial output of objdump -s:
>>>> > 04f0 00030000 5f6d6f76 655f636f 6e737472 ...._move_constr
>>>> > 0500 7563745f 66776b2e 68000300 00474454 uct_fwk.h....GDT
>>>> > 0510 79706573 2e68000a 00007365 72646566 ypes.h....serdef
>>>> > 0520 732e6800 01000073 75666669 782e6870 s.h....suffix.hp
>>>> > 0530 70000b00 00646973 74636364 5f616333 p....distccd_ac3
>>>> > 0540 31633936 612e6969 000c0000 61646c5f 1c96a.ii....adl_
>>>> > 0550 62617272 6965722e 68707000 0d000062 barrier.hpp....b
>>>> > 0560 6f6f6c5f 6677642e 68707000 0e000069 ool_fwd.hpp....i
>>>> > 0570 6e746567 72616c5f 635f7461 672e6870 ntegral_c_tag.hp
>>>> > 0580 70000e00 00766f69 645f6677 642e6870 p....void_fwd.hp
>>>> >
>>>> > Thank you for reviewing my issue.
>>>> >
>>>> > -Jeff
>>>> >
>>>> > __
>>>> > distcc mailing list http://distcc.samba.org/
>>>> > To unsubscribe or change options:
>>>> > https://lists.samba.org/mailman/listinfo/distcc
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Martin
>>>>
>>>
>>>
>>> __
>>> distcc mailing list http://distcc.samba.org/
>>> To unsubscribe or change options:
>>> https://lists.samba.org/mailman/listinfo/distcc
>>>
>>
>>
>>
>> --
>> Don't walk behind me, I may not lead.
>> Don't walk in front of me, I may not follow.
>> Just walk beside me and be my friend.
>> -- Albert Camus (attributed to)
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/distcc/attachments/20100629/4622f65b/attachment-0001.html>