[distcc] homogeneous environments

[Robert W. Anderson]

Fergus Henderson wrote:
> The include server could be the bottleneck.  What's the CPU usage for 
> the include server process?
> Or it could be disk I/O.  Try iostat or vmstat to profile that.

After some more experimentation, I think I may have a clue what's going 
on here.  I think I may be getting bound up in context switching costs, 
both in my single node performance and on localhost using distcc.

On a single node:

-j4: 24 million context switches @ 10us = 4m
-j8: 96 million context switches @ 10us = 16m

         Command being timed: "make -j4 debug"
         User time (seconds): 378.81
         System time (seconds): 566.57
         Percent of CPU this job got: 265%
         Elapsed (wall clock) time (h:mm:ss or m:ss): 5:55.91
         Major (requiring I/O) page faults: 472
         Minor (reclaiming a frame) page faults: 26761810
         Voluntary context switches: 24629071
         Involuntary context switches: 269418

         Command being timed: "make -j8 debug"
         User time (seconds): 427.55
         System time (seconds): 2250.72
         Percent of CPU this job got: 429%
         Elapsed (wall clock) time (h:mm:ss or m:ss): 10:23.29
         Major (requiring I/O) page faults: 270
         Minor (reclaiming a frame) page faults: 26760678
         Voluntary context switches: 95391888
         Involuntary context switches: 825155

I don't understand why doubling the job count increases context 
switching by 4x.  Any insights appreciated.

Thanks,
-- 
Robert W. Anderson
Center for Applied Scientific Computing
Email: anderson110 at llnl.gov
Tel: 925-424-2858  Fax: 925-423-8704