rsync vs. rcp
jw at pegasys.ws
Thu Feb 20 13:20:02 EST 2003
On Thu, Feb 20, 2003 at 01:03:16PM +1100, Donovan Baarda wrote:
> On Thu, 2003-02-20 at 11:36, Craig Barratt wrote:
> > > RSYNC DOES NOT WORK WITH 1GB+ FILES... unless you have a sufficiently
> > > large block size. See the following;
> > >
> > > http://firstname.lastname@example.org/msg05219.html
> > Let's be careful here. Rsync *does* work on 1GB+ files. What you
> > probably meant to say was that rsync's first pass might fail on
> > files that have 1GB+ of changes. But the second pass (which uses
> > the full MD4 block checksum) will work correctly.
> Yeah, I was referring to the rsync algorithm as implemented in rsync,
> not the rsync tool as a whole. Even then, it arguably does not work in
> terms of it ends up transferring much more data (up to 2x) than a pure
> scp does. It does end up correctly transferring the file though.
> > So a more correct statement is that Rsync might work *slowly* on
> > files with 1GB+ of changes because two passes are required.
> > BTW, rsync already has an adaptive block size unless you set it
> > on the command-line. The block size is roughly
> > min(16384, max(700, file_size/10000))
> > ie: file_size/10000, but no less than 700 and no more than 16384.
> I wasn't aware that it had this. Was it there at the time of the
> original discussion (Oct 2002)? The people involved in the discussion
> then didn't seem to know this.
> However, it's not really adequate. A 16K block size only really works
> for files up to about 500M. Still... that's a lot better than I thought
> it was at the time.
I'd be very open to increasing the upper limit on the dynamic
block size but it should be done in balance with increasing
(dynamically per-file) the block csum-length.
J.W. Schultz Pegasystems Technologies
email address: jw at pegasys.ws
Remember Cernan and Schmitt
More information about the rsync