[distcc] homogeneous environments

[Robert W. Anderson]

Fergus Henderson wrote:
>     The plot thickens a bit.  I had been using the eth0 address in my
>     --allow option, but the address I see here is related to an
>     InfiniBand interconnect, according to ifconfig.  If I try allowing
>     the address 192.168.120.3, it reports the same "denied" error.  If I
>     use the full address as shown with the colon suffix, I get an error
>     "can't parse internet address".
> 
>     Any ideas?
> 
> 
> What's the distccd log output if you start distccd with --verbose?

If you're truly interested, I'll try it.  But in the meantime I just 
disabled the client check in the source and everything seemed to work 
from that point.

What's possibly more interesting is the performance results.  I'm 
compiling about 1300 source files on nodes that have 16 cpu's each.

Single node (plain GNU make, no distcc):

-j2 8m 19s
-j4 5m 46s
-j5 6m 39s
-j8 10m 35s

I don't understand this, but it is repeatable.  Any ideas on that one? 
I'm not sure how to effectively profile this. All the sources are on NFS.

So I then went multi-node w/ 4 jobs per node.  Using localhost as a 
server only seems to slow things down, incidentally.

1 node,  -j4:  5m 28s (using distcc and 1 remote node)
2 nodes, -j8:  2m 57s
3 nodes, -j12: 2m 16s
4 nodes, -j16: 1m 58s
5 nodes, -j20: 2m 7s

Scaling seems to break down around the 4 node mark.  Our link step is 
only 5-6 seconds, so we are not getting bound by that.  Messing with -j 
further doesn't seem to help.  Any ideas for profiling this to find any 
final bottlenecks?

Thanks,
-- 
Robert W. Anderson
Center for Applied Scientific Computing
Email: anderson110 at llnl.gov
Tel: 925-424-2858  Fax: 925-423-8704