Reliability and robustness problems

[John]

Hmm
I subscribed to the list before I sent this. I've not seen either the 
email confirmation request or my mail to the list.

Wayne Davison wrote:

>On Tue, Jun 08, 2004 at 07:37:32AM +0800, John wrote:
>  
>
>>1. ADSL connexion at one end ot the other dropping for a while. rsync 
>>doesn't notice and mostly  hangs. I have seen rsync at home still 
>>running but with no relevant files open.
>>    
>>
>
>There are two aspects of this: (1) Your remote shell should be setup to
>timeout appropriately (which is why rsync doesn't timeout by default) --
>see your remote-shell's docs for how to do this; (2) you can tell rsync
>to timeout after a certain amount of inactivity (see --timeout).
>
>  
>

I'm pretty sure that ssh times out properly: certainly it disconnects 
some time after my dialup line goes down.

I'd managed to overlook the --timeout option on rsync.

I'll look more closely for ssh hanging round next time it happens, but 
I'm skeptical.

>>2. rsync uses an enormous amount of virtual memory
>>    
>>
>
>Yes, it uses something like 80-100 bytes or so per file in the
>transferred hierarchy (depending on options) plus a certain base amount
>of memory.  Your options are to (1) copy smaller sections of the
>hierarchy at a time, (2) add more memory, or (3) help code something
>better.  This is one of the big areas that I've wanted to solve by
>completely replacing the current rsync protocol with something better
>(as I did in my rZync testbed protocol project a while back -- it
>transfers the hierarchy incrementally, so it never has more than a
>handful of directories in action at any one time).  At some point I will
>get back to working on an rsync-replacement project.
>  
>
1. No chance, I'd think, of it handling hard links properly if it can't 
see at least one of the other "copies." However, I could easily be wrong.
2. May require new boxes all round. Money may be an issue.
3. Yeah. My C skills a pretty rudimentary; I can barely read the stuff. 
Time for me to learn it was 30 or even years go, but I don't think it 
was invented then.

>  
>
>>3. rsync does not detect when its partner has vanished.
>>    
>>
>
>That seems unlikely unless the remote shell is still around.  If the
>shell has terminated, the socket would return an EOF and rsync would
>exit.  So, I'll assume (until shown otherwise) that this is a case of
>the remote shell still hanging around.
>
>  
>
I've been known to have "unlikely" failures before:-) It could be the 
bloody Billion in the way. I know that if a TCP session is quiet "too 
long" the Billion forgets all about it.

The fact I'm now trying a VPN using UDP should overcome that issue.


>>3a. It'd like to see rsync have the ability to retry in the case it's 
>>initiated the transfer.
>>    
>>
>
>There has been some talk of this recently.  It doesn't seem like it
>would be too hard to do, but it's not trivial either.  If someone wanted
>to code something up, I'd certainly appreciate the assistance.  Or feel
>free to put an enhancement request into bugzilla.  (BTW:  has anyone
>heard from J.W. Schultz anytime recently?  He seems to have dropped off
>the net without any explanation about 3 months ago -- I hope he's OK.)
>
>  
>
>>4. [...] As best I can tell, rsync is transferring each copy fresh
>>instead of recognising the hard link before the transfer and getting
>>the destination rsync to make a new hard link.
>>    
>>
>
>This should not be the case if you use the -H option.  (It also helps
>to use 2.6.2 on both ends, as the memory-consumption was reduced
>considerably from older releases.)  If you're seeing a problem with
>this, you should provide full details on what command you're running,
>what versions you're using, and as small a test case as you can that
>shows the problem.
>  
>
Well, I cut and pasted the commandline (with only a minor edit to 
disguise relevant sites). Here 'tis again:
rsync --recursive --links --hard-links --perms --owner --group --devices 
--times --sparse --one-file-system --rsh=/usr/bin/ssh --delete 
--delete-excluded --delete-after --max-delete=80 --relative --stats 
--numeric-ids

At the moment I'm running the standard latest versions for Woody 
(office) and Red Hat Linux 9 (home).

Acutally, Woody may be one fix behind latest, there as a new rsync out 
in the past few days that went in this morning. Nah, looks like the 
update went in before the last started, the rync binary it's got open 
isn't "deleted."
Office:
rsync  version 2.5.6cvs  protocol version 26
Home:
rsync  version 2.5.7  protocol version 26

Is there a source of pre-built binaries? I didn't see it on the Rsync site.

>..wayne..
>
>  
>