patch to enable faster mirroring of large filesystems

Andrew J. Schorr ajschorr at yahoo.com
Tue Nov 27 01:41:22 EST 2001


On Sun, Nov 25, 2001 at 03:21:51AM +1100, Martin Pool wrote:
> On 20 Nov 2001, Dave Dykstra <dwd at bell-labs.com> wrote:
> 
> > > And, by the way, even if the batch stuff accomplishes the same performance
> > > gains, I would still argue that the "--files-from" type of behavior
> > > that I implemented is a nice transparent interface that people might
> > > like to have.  The ability to pipe in output from gfind -print0 opens
> > > up some possibilities.
> > 
> > Yes, many people have argued that a --files-from type of option is
> > desirable and I agree.
> 
> I agree too.  I think there would certainly be no argument with taking
> a patch that did only that.  (Except that it makes the option list
> even more ridiculously bloated.)
> 
> I think a better fix is to transfer directories one by one, pipelined,
> rather than walking the whole tree upfront.  This will take a protocol
> change and a fair amount of work, but I think it is possible.  If we
> can get it to just work faster transparently that is better.

I understand your point of view, but I think it is a mistake to
hold rsync's algorithm hostage to the directory tree traversal logic
built into the program.  IMHO, the basic file transfer algorithm of
rsync is terrific, but the program wrapped around it is a bit out of
control.

The spirit of my patch is to expose the low-level rsync algorithm and
to allow people to build up their customized infrastructure outside
of the program instead of having to build it in.  I think this is in
the spirit of Unix tools.  I think if rsync were to expose some of its
low-level capabilities, then we would not have a need for xdelta and rdiff,
projects which are springing up because of rsync's opaqueness.

Anyway, you may not like the way my patch is implemented, but I still argue
that it serves a useful purpose, and it gets the job done for me.

Cheers,
Andy




More information about the rsync mailing list