rsync is slowing down

[Wayne Davison]

On Wed, Apr 07, 2004 at 06:44:35PM -0500, Phil Howard wrote:
> How would the sending side know what directories are "old" for a
> given receiver?

I didn't see anything about multiple receivers in your initial email,
but it's not hard to come up with an optimizing strategy that allows
more than a single one.  [For instance, maintain a summary file for each
archive-hierarchy delineation that exists, and if any of the summary
files get updated during the initial slurp of data (which would exclude
all archived data but not the summary files), start a second transfer
that updates the relevant archive dir(s).]

The big question is, should rsync get some kind of optimizing algorithm
that expects/updates external data for the purposes of optimizing the
filesystem search, or should this be something that exists outside of
rsync.  The current rsync algorithm is not made for the kind of short-
circuiting of directory descent that this optimization would require, so
I would not attempt to shoe-horn it into it.  Something like this would
work a lot better with a more modern protocol that processed each
directory in succession rather than the whole tree all at once.  There
has been a good bit of talk (and even a little work) on designing an
rsync successor, but nothing is forthcoming at the moment.

..wayne..