Transfering very large files / and restarting failures

[Wayne Davison]

On Wed, Jul 27, 2005 at 04:29:46PM -0700, Todd Papaioannou wrote:
> Not sure I have the mojo to mess with the patches though! 

I applied the --append patch to the CVS source, so if you want to snag
version 2.6.7cvs, you can grab it via the latest "nightly" tar file:

    http://rsync.samba.org/ftp/rsync/nightly/

I did some very simple timing tests to see how fast it would be to do a
local transfer of a 250MB file that had a little over the first half of
the file already present.  The results were:

Normal --whole-file:      ~31 seconds  (straight copy, data speedup 1.00)
Forced --no-whole-file:   ~73 seconds  (data speedup 2.39)
Using --inplace:          ~30 seconds  (data speedup 2.39)
Using --update:           ~24 seconds  (data speedup 2.40)

(The data speedup values are rsync's standard "speedup" values, which
indicate how much the transferred data was reduced over the "wire".)

Also keep in mind that even though the --append option is writing out
less than 1/2 the file, it is still reading in all the existing data in
the partial file in order to compute the full-file checksum.

I then tried the same transfer test between two systems on my local
wireless network (11g).  Here are the results:

Forced --whole-file:     ~134 seconds  (straight copy, data speedup 1.00)
Normal --no-whole-file:  ~100 seconds  (data speedup 2.39)
Using --inplace:          ~95 seconds  (data speedup 2.39)
Using --update:           ~60 seconds  (data speedup 2.40)

> is there another protocol you might know of, other than ftp that
> supports byte level restart/append? 

I can recall seeing the continue feature in wget and bittorrent.

..wayne..