Reliability and robustness problems

[Wayne Davison]

On Tue, Jun 08, 2004 at 07:37:32AM +0800, John wrote:
> 1. ADSL connexion at one end ot the other dropping for a while. rsync 
> doesn't notice and mostly  hangs. I have seen rsync at home still 
> running but with no relevant files open.

There are two aspects of this: (1) Your remote shell should be setup to
timeout appropriately (which is why rsync doesn't timeout by default) --
see your remote-shell's docs for how to do this; (2) you can tell rsync
to timeout after a certain amount of inactivity (see --timeout).

> 2. rsync uses an enormous amount of virtual memory

Yes, it uses something like 80-100 bytes or so per file in the
transferred hierarchy (depending on options) plus a certain base amount
of memory.  Your options are to (1) copy smaller sections of the
hierarchy at a time, (2) add more memory, or (3) help code something
better.  This is one of the big areas that I've wanted to solve by
completely replacing the current rsync protocol with something better
(as I did in my rZync testbed protocol project a while back -- it
transfers the hierarchy incrementally, so it never has more than a
handful of directories in action at any one time).  At some point I will
get back to working on an rsync-replacement project.

> 3. rsync does not detect when its partner has vanished.

That seems unlikely unless the remote shell is still around.  If the
shell has terminated, the socket would return an EOF and rsync would
exit.  So, I'll assume (until shown otherwise) that this is a case of
the remote shell still hanging around.

> 3a. It'd like to see rsync have the ability to retry in the case it's 
> initiated the transfer.

There has been some talk of this recently.  It doesn't seem like it
would be too hard to do, but it's not trivial either.  If someone wanted
to code something up, I'd certainly appreciate the assistance.  Or feel
free to put an enhancement request into bugzilla.  (BTW:  has anyone
heard from J.W. Schultz anytime recently?  He seems to have dropped off
the net without any explanation about 3 months ago -- I hope he's OK.)

> 4. [...] As best I can tell, rsync is transferring each copy fresh
> instead of recognising the hard link before the transfer and getting
> the destination rsync to make a new hard link.

This should not be the case if you use the -H option.  (It also helps
to use 2.6.2 on both ends, as the memory-consumption was reduced
considerably from older releases.)  If you're seeing a problem with
this, you should provide full details on what command you're running,
what versions you're using, and as small a test case as you can that
shows the problem.

..wayne..