efficient file appends

[David Bolen]

rsync at ka9q.net [rsync at ka9q.net] writes:

> >While potentially a useful option, you wouldn't want the protocol to
> >automatically always check for it, since it would preclude rsync on
>
> This extension need not break any existing mechanism; if the hash of
> the receiver's copy of the file doesn't match the start of the
> sender's file, the protocol would continue as before.

Well, my point was that even if it does match, you might still want
the protocol to continue as before.  For example, if you have a file
that grows, but tends to contain similar information.  In that case,
you still want the per-block checksum information from the destination
because that way the source can use that information to minimize the
amount of new information to transmit.  Without having the per-block
information, it can't tell how to extract data from the current copy
at the destination to re-use for the new data rather than sending the
new data directly.  Not a big deal for appending log files (as long as
they have changing date strings), but not necessarily something to
have enabled by default.

> >Alternatively, even with rsync the way it is today, what I do is
> >manually bump up the blocksize to something large (say 16 or 32K).
> 
> This sounds like an excellent idea, and I'll give it a try. As the
> blocksize reaches the receiver's file size, the scheme essentially
> approaches my idea.

Hmm, I've never tried _really_ large block sizes (I thought I had
problems if I got close to 64K, but I may be mis-remembering).  The
one drawback to the larger block sizes is that if you do encounter any
differences, you'll retransmit more information than necessary, but if
you do beforehand it's definitely just appended dat that won't be the
case.

-- David

/-----------------------------------------------------------------------\
 \               David Bolen            \   E-mail: db3l at fitlinxx.com  /
  |             FitLinxx, Inc.            \  Phone: (203) 708-5192    |
 /  860 Canal Street, Stamford, CT  06902   \  Fax: (203) 316-5150     \
\-----------------------------------------------------------------------/