[distcc] Using distcc on two ARM boards resulted in almost no acceleration...

[James W Mills]

On Aug 4, 2010, at 9:26 PM, Martin Pool wrote:

> On 3 August 2010 04:07, James W Mills <jameswmills at ti.com> wrote:
>> Hello list!
>> 
>> First of all, thanks for this great tool.  If I can get it to work for me, it will really help!
>> 
>> Okay, I have two OMAP4430 Panda boards, both running a 2.6.34 kernel and Ubuntu 10.10 Maverick.  They are running distcc version 3.1, and gcc version 4.4.4, both native ports by Ubuntu for the arm.
>> 
>> These two machines are connected to each other via ethernet on a private switched network.  They have an ~50Mb/s link.  I have configured distccd on the host machine (192.168.1.20) to listen to all interfaces (0.0.0.0), and to allow connections from "127.0.0.1 192.168.1.0/24".  The remote machine (192.168.1.10) is configured to allow "127.0.0.1 192.168.1.0/24" and to listen on 192.168.1.10.
>> 
>> I am building a 2.6.34 kernel (make uImage).  I have built 4 times, twice locally, and twice "distributed".  Here are my results:
>> 
>> make clean
>> time make CC="distcc gcc" DISTCC_HOSTS="localhost 192.168.1.10" -j8 uImage
>> real    44m44.648s
>> user    30m4.039s
>> sys     4m0.313s
>> 
>> make clean
>> time make CC="distcc gcc" DISTCC_HOSTS="localhost 192.168.1.10" -j4 uImage
>> real    42m32.055s
>> user    33m46.500s
>> sys     3m55.273s
>> 
>> make clean
>> time make -j4 uImage
>> real    45m19.444s
>> user    57m2.945s
>> sys     3m54.578s
>> 
>> make clean
>> time make -j2 uImage
>> real    45m10.431s
>> user    54m29.516s
>> sys     3m56.813s
>> 
>> 
>> As you can see, I saw no "real" speedup by using distcc.  Is there something I am doing wrong?
> 
> Hi James,
> 
> The first thing to do would be to set DISTCC_VERBOSE=1 and
> DISTCC_LOG=something and look in the distcc log to see if it's
> actually trying to distribute the jobs or  if not, why not.
> 
> I think the problem may be that setting DISTCC_HOSTS as an argument to
> make only sets the make internal variable, not the environment
> variable, so distcc won't see it.  Try putting it at the start of the
> command line.

Thanks Martin, those are great hints.

I ran the job again, and this time bounced between the two boxes using 'lsof -i -P', and grep'ing for connected distccd processes.  The distcc process seems to be working, as I saw anywhere from zero to twelve remote connections between the two machines.

From an active CPU standpoint, the compile times are certainly going down.  Unfortunately, the actual elapsed time is holding relatively fast.  Could the problem be attributed to my slow(er) network? 

I will set DISTCC_VERBOSE and gather up further information.

Thanks again!
~james