Rsync connection times out on very large --files-from

[Eli]

Wayne wrote:
> Since rsync is keeping the socket busy during the building
> of the list (it is trasmitting it during that time), I
> guess the sorting/cleaning of the list is taking enough
> time that the socket timeouted out on you.

Interesting, I didn't know rsync would transmit the list to the rsync server
as it processes it.  The list of files is just over 1million files, and
takes find several hours to generate but that's irrelevant since rsync isn't
called until after the list is generated.  It does take rsync a while to
parse the list of files before it gets to sending file data.

> I'm glad that the setting of SO_KEEPALIVE did something to 
> help fix your problem, but I'm having a hard time seeing why:
> the code should set that for all daemon socket connections
> (excluding only daemon-over-ssh).
> Are you using the --timeout=N option to rsync (which is an 
> internal-to-rsync timeout value, unrelated to the TCP timeout).

No, I'm not using any --timeout setting on the client or server.  There
could be another reason why the rsync worked this time... I needed to back
up the files regardless, so I wrote a PHP parser script to split up the huge
list of files in to groups of files based on parent folder (one list of
files per parent) which resulted in 8109 separate lists of files.  I then
used a for loop in the shell to iterate through each of those 8109 files and
run rsync on them.  Doing this allowed the rsync processes to not time out
their connections and they copied all their data (after MANY hours - it was
close to 200gb of data, and I didn't use -z).  Once this data was copied to
the rsync server, I *THEN* tested the SO_KEEPALIVE setting in the
rsyncd.conf for subsequent runs of the original command (not splitting up
the list of files this time).  The rsync process then did not error out and
was able to update the files fine.  This could very well be due to rsync
taking far less time to do its task since it just had to compare files on
the rsync server and not transmit as many files (an update compared to a 1st
time back up run).

I don't have any debugger programs installed on the client system so I can't
run rsync in a debugger (not to mention the output would probably run my
system out of space!), but let me know if there's any tests you want me to
do.  Shall I delete all the files on the rsync server and re-try the rsync
line to see if it still times out with SO_KEEPALIVE enabled?

Eli.