rsync vs. rcp

Donovan Baarda abo at minkirri.apana.org.au
Thu Feb 20 13:03:16 EST 2003


On Thu, 2003-02-20 at 11:36, Craig Barratt wrote:
> > RSYNC DOES NOT WORK WITH 1GB+ FILES... unless you have a sufficiently
> > large block size. See the following;
> > 
> > http://www.mail-archive.com/rsync@lists.samba.org/msg05219.html
> 
> Let's be careful here.  Rsync *does* work on 1GB+ files.  What you
> probably meant to say was that rsync's first pass might fail on
> files that have 1GB+ of changes.  But the second pass (which uses
> the full MD4 block checksum) will work correctly.

Yeah, I was referring to the rsync algorithm as implemented in rsync,
not the rsync tool as a whole. Even then, it arguably does not work in
terms of it ends up transferring much more data (up to 2x) than a pure
scp does. It does end up correctly transferring the file though.
 
> So a more correct statement is that Rsync might work *slowly* on
> files with 1GB+ of changes because two passes are required.
> 
> BTW, rsync already has an adaptive block size unless you set it
> on the command-line.  The block size is roughly
> 
>     min(16384, max(700, file_size/10000))
> 
> ie: file_size/10000, but no less than 700 and no more than 16384.

I wasn't aware that it had this. Was it there at the time of the
original discussion (Oct 2002)? The people involved in the discussion
then didn't seem to know this.

However, it's not really adequate. A 16K block size only really works
for files up to about 500M. Still... that's a lot better than I thought
it was at the time.

-- 
----------------------------------------------------------------------
ABO: finger abo at minkirri.apana.org.au for more info, including pgp key
----------------------------------------------------------------------



More information about the rsync mailing list