[distcc] distcc 1.2.3 fragility?

[Wayne Davison]

So, no one has anything to say about how version 1.2.3 is working for
them?  I did some more 3-machine compiles yesterday to try to figure out
what was going wrong, and unfortunately, I didn't manage to answer any
of my questions.

I first fell back to version 1.2.1 on all 3 machines and then tried to
get a compile of the phoenix web browser to fail.  Two compiles later
and nary a failure in sight, I decided to go back to version 1.2.3 and
see if it was still failing.  One successful compile later and I was
very baffled (side note:  yes I made sure that the compiles weren't
using my ccache dir to avoid calling distcc).  Note that no retry
warnings occurred on any of these 3 new compiles, so I have no idea
what was causing the original failures, or why it was now working.

Attached is the distccd log messages from one of the systems back when
the compile failed.  It looks like the sys_sendfile() returned a partial
write after about 25 minutes of waiting around, and then, when the code
called sys_sendfile to write the remainder, it returned an ETIMEDOUT
error.  Anyone have any ideas what might have gone wrong?  It seems
strange that the distcc side of the socket didn't get an error reading
the data when the distccd side finally gave up and closed the socket.

..wayne..
-------------- next part --------------
Mar  4 16:38:58 miketv distccd[27326]: (dcc_reap_kids) down to 0 children
Mar  4 16:38:58 miketv distccd[27326]: (dcc_reap_kids) Notice: child 29420 exited: exit status 0
Mar  4 16:38:58 miketv distccd[27326]: (dcc_parent_loop) waiting to accept connection
Mar  4 16:39:09 miketv distccd[27326]: (dcc_serve_connection) up to 1 children
Mar  4 16:39:09 miketv distccd[27326]: (dcc_parent_loop) waiting to accept connection
Mar  4 16:39:09 miketv distccd[29424]: (dcc_check_client) connection from 192.168.0.6
Mar  4 16:39:09 miketv distccd[29424]: (dcc_r_argv) got arguments: g++ -c -fPIC -frtti -fno-exceptions -Wconversion -Wpointer-arith -Wcast-align -Woverloaded-virtual -Wsynth -Wno-ctor-dtor-privacy -Wno-long-long -O2 -mcpu=i686 -pipe -s -fforce-addr -Wno-deprecated -fshort-wchar -pthread -pipe -ffunction-sections -o /home/root/.ccache/tmp.hash.dot.blorf.net.8839.o /home/root/.ccache/tmp.stdout.dot.blorf.net.8839.ii
Mar  4 16:39:09 miketv distccd[29424]: compile from /home/root/.ccache/tmp.stdout.dot.blorf.net.8839.ii to /home/root/.ccache/tmp.hash.dot.blorf.net.8839.o
Mar  4 16:39:09 miketv distccd[29424]: (dcc_accept_job) input file /home/root/.ccache/tmp.stdout.dot.blorf.net.8839.ii, output file /home/root/.ccache/tmp.hash.dot.blorf.net.8839.o
Mar  4 16:39:09 miketv distccd[29424]: (dcc_r_file) received 300349 bytes to file /tmp/distcc_000003ea/server_0000029424.ii
Mar  4 16:39:09 miketv distccd[29424]: (dcc_r_file_timed) 300349 bytes received in 0.025s, rate 11798kB/s
Mar  4 16:39:10 miketv distccd[27326]: (dcc_serve_connection) up to 2 children
Mar  4 16:39:10 miketv distccd[29428]: (dcc_check_client) connection from 192.168.0.6
Mar  4 16:39:10 miketv distccd[29428]: (dcc_r_argv) got arguments: g++ -c -fPIC -frtti -fno-exceptions -Wconversion -Wpointer-arith -Wcast-align -Woverloaded-virtual -Wsynth -Wno-ctor-dtor-privacy -Wno-long-long -O2 -mcpu=i686 -pipe -s -fforce-addr -Wno-deprecated -fshort-wchar -pthread -pipe -ffunction-sections -o /home/root/.ccache/tmp.hash.dot.blorf.net.8848.o /home/root/.ccache/tmp.stdout.dot.blorf.net.8848.ii
Mar  4 16:39:10 miketv distccd[29428]: compile from /home/root/.ccache/tmp.stdout.dot.blorf.net.8848.ii to /home/root/.ccache/tmp.hash.dot.blorf.net.8848.o
Mar  4 16:39:10 miketv distccd[29428]: (dcc_accept_job) input file /home/root/.ccache/tmp.stdout.dot.blorf.net.8848.ii, output file /home/root/.ccache/tmp.hash.dot.blorf.net.8848.o
Mar  4 16:39:10 miketv distccd[27326]: (dcc_parent_loop) waiting to accept connection
Mar  4 16:39:10 miketv distccd[29428]: (dcc_r_file) received 254391 bytes to file /tmp/distcc_000003ea/server_0000029428.ii
Mar  4 16:39:10 miketv distccd[29428]: (dcc_r_file_timed) 254391 bytes received in 0.060s, rate 4140kB/s
Mar  4 16:39:14 miketv distccd[29428]: (dcc_x_file_timed) 25340 bytes sent in 0.002s, rate 15752kB/s
Mar  4 16:39:14 miketv distccd[29428]: (dcc_report_rusage) g++ resource usage: 1.880000s user, 0.040000s system
Mar  4 16:39:14 miketv distccd[29428]: g++ on miketv.blorf.net completed ok
Mar  4 16:39:14 miketv distccd[29428]: (dcc_accept_job) complete; output file: 25340 bytes
Mar  4 16:39:14 miketv distccd[29428]: (dcc_exit) exit: code 0; self: 0.0000 user 0.10000 sys; children: 1.880000 user 0.40000 sys
Mar  4 16:39:14 miketv distccd[27326]: (dcc_reap_kids) down to 1 children
Mar  4 16:39:14 miketv distccd[27326]: (dcc_reap_kids) Notice: child 29428 exited: exit status 0
Mar  4 16:39:14 miketv distccd[27326]: (dcc_parent_loop) waiting to accept connection
Mar  4 16:39:14 miketv distccd[27326]: (dcc_serve_connection) up to 2 children
Mar  4 16:39:14 miketv distccd[27326]: (dcc_parent_loop) waiting to accept connection
Mar  4 16:39:14 miketv distccd[29432]: (dcc_check_client) connection from 192.168.0.6
Mar  4 16:39:14 miketv distccd[29432]: (dcc_r_argv) got arguments: g++ -c -fPIC -frtti -fno-exceptions -Wconversion -Wpointer-arith -Wcast-align -Woverloaded-virtual -Wsynth -Wno-ctor-dtor-privacy -Wno-long-long -O2 -mcpu=i686 -pipe -s -fforce-addr -Wno-deprecated -fshort-wchar -pthread -pipe -ffunction-sections -o /home/root/.ccache/tmp.hash.dot.blorf.net.8877.o /home/root/.ccache/tmp.stdout.dot.blorf.net.8877.ii
Mar  4 16:39:14 miketv distccd[29432]: compile from /home/root/.ccache/tmp.stdout.dot.blorf.net.8877.ii to /home/root/.ccache/tmp.hash.dot.blorf.net.8877.o
Mar  4 16:39:14 miketv distccd[29432]: (dcc_accept_job) input file /home/root/.ccache/tmp.stdout.dot.blorf.net.8877.ii, output file /home/root/.ccache/tmp.hash.dot.blorf.net.8877.o
Mar  4 16:39:14 miketv distccd[29432]: (dcc_r_file) received 209096 bytes to file /tmp/distcc_000003ea/server_0000029432.ii
Mar  4 16:39:14 miketv distccd[29432]: (dcc_r_file_timed) 209096 bytes received in 0.013s, rate 15939kB/s
Mar  4 16:39:17 miketv distccd[29432]: (dcc_x_file_timed) 14084 bytes sent in 0.000s, rate 78594kB/s
Mar  4 16:39:17 miketv distccd[29432]: (dcc_report_rusage) g++ resource usage: 0.990000s user, 0.030000s system
Mar  4 16:39:17 miketv distccd[29432]: g++ on miketv.blorf.net completed ok
Mar  4 16:39:17 miketv distccd[29432]: (dcc_accept_job) complete; output file: 14084 bytes
Mar  4 16:39:17 miketv distccd[29432]: (dcc_exit) exit: code 0; self: 0.0000 user 0.10000 sys; children: 0.990000 user 0.30000 sys
Mar  4 16:39:17 miketv distccd[27326]: (dcc_reap_kids) down to 1 children
Mar  4 16:39:17 miketv distccd[27326]: (dcc_reap_kids) Notice: child 29432 exited: exit status 0
Mar  4 16:39:17 miketv distccd[27326]: (dcc_parent_loop) waiting to accept connection
Mar  4 16:54:52 miketv distccd[29424]: (dcc_pump_sendfile) Notice: sendfile: partial transmission of 37431 bytes; retrying 13529 @37431
Mar  4 16:54:52 miketv distccd[29424]: (dcc_pump_sendfile) ERROR: sendfile failed: Connection timed out
Mar  4 16:54:52 miketv distccd[29424]: (dcc_x_file_timed) 50960 bytes sent in 934.730s, rate 0kB/s
Mar  4 16:54:52 miketv distccd[29424]: (dcc_report_rusage) g++ resource usage: 4.770000s user, 0.050000s system
Mar  4 16:54:52 miketv distccd[29424]: g++ on miketv.blorf.net completed ok
Mar  4 16:54:52 miketv distccd[29424]: (dcc_accept_job) complete; output file: 50960 bytes
Mar  4 16:54:52 miketv distccd[29424]: (dcc_exit) exit: code -1; self: 0.10000 user 0.10000 sys; children: 4.770000 user 0.50000 sys
Mar  4 16:54:52 miketv distccd[27326]: (dcc_reap_kids) down to 0 children
Mar  4 16:54:52 miketv distccd[27326]: (dcc_reap_kids) Notice: child 29424 exited: exit status 255
Mar  4 16:54:52 miketv distccd[27326]: (dcc_parent_loop) waiting to accept connection