rsync vs. rcp

[jw schultz]

On Thu, Feb 20, 2003 at 01:03:16PM +1100, Donovan Baarda wrote:
> On Thu, 2003-02-20 at 11:36, Craig Barratt wrote:
> > > RSYNC DOES NOT WORK WITH 1GB+ FILES... unless you have a sufficiently
> > > large block size. See the following;
> > > 
> > > http://www.mail-archive.com/rsync@lists.samba.org/msg05219.html
> > 
> > Let's be careful here.  Rsync *does* work on 1GB+ files.  What you
> > probably meant to say was that rsync's first pass might fail on
> > files that have 1GB+ of changes.  But the second pass (which uses
> > the full MD4 block checksum) will work correctly.
> 
> Yeah, I was referring to the rsync algorithm as implemented in rsync,
> not the rsync tool as a whole. Even then, it arguably does not work in
> terms of it ends up transferring much more data (up to 2x) than a pure
> scp does. It does end up correctly transferring the file though.
>  
> > So a more correct statement is that Rsync might work *slowly* on
> > files with 1GB+ of changes because two passes are required.
> > 
> > BTW, rsync already has an adaptive block size unless you set it
> > on the command-line.  The block size is roughly
> > 
> >     min(16384, max(700, file_size/10000))
> > 
> > ie: file_size/10000, but no less than 700 and no more than 16384.
> 
> I wasn't aware that it had this. Was it there at the time of the
> original discussion (Oct 2002)? The people involved in the discussion
> then didn't seem to know this.
> 
> However, it's not really adequate. A 16K block size only really works
> for files up to about 500M. Still... that's a lot better than I thought
> it was at the time.

I'd be very open to increasing the upper limit on the dynamic
block size but it should be done in balance with increasing
(dynamically per-file) the block csum-length.

-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw at pegasys.ws

		Remember Cernan and Schmitt