rsync in-place (was Re: rsync 1tb+ each day)

[jw schultz]

On Tue, Feb 04, 2003 at 02:37:26PM -0500, Bennett Todd wrote:
> 2003-02-04T14:29:48 Kenny Gorman:
> > Is it possible to tell rsync to update the blocks of the target file 
> > 'in-place' without creating the temp file (the 'dot file')?  I can 
> > guarantee that no other operations are being performed on the file at 
> > the same time.  The docs don't seem to indicate such an option.
> 
> No, it's not possible, and making it possible would require a deep
> and fundamental redesign and re-implementation of rsync; the result
> wouldn't resemble the current program much.
> 
> Here's a sketch of the heart of the rsync algorithm (for finer
> details, see the tech report available from[1]).
> 
> Let's call the two endpoints the sender (who has the newer version
> of the file), and the receiver (who wants to update its older local
> copy to match that on the sender).
> 
> The receiver computes checksums on each block of the destination
> file, and streams them to the sender.
> 
> The sender finds all instances of any of those blocks in the source
> file. Then the sender transmits instructions to the receiver,
> describing how to build a spiffy new copy of the newer source file,
> using a mixture of actual chunks of new contents, and blocks taken
> from the older version of the file. The receiver follows these
> instructions, copying blocks as needed from the old version and
> combining them with the new bits to construct the new file. It's
> then moved into place.
> 
> This algorithm by nature expects that the old version of the
> destination file is used as a source for taking blocks, in building
> the new version. Adjusting this algorithm to work in-place is
> non-trivial.

The reason why in-place updating is difficult is that
rsync expects the unchanged blocks in the old file may be
relocated.  Data inserted into or removed from the file does
not require the rest of the file to be retransmitted.
Unchanged blocks will be copied from the old locations in
the old file to new locations in the new file.

In-place updates requires that blocks not relocate.
It may be possible by disallowing matches having differing
offsets.  That would require deeper investigation.


-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw at pegasys.ws

		Remember Cernan and Schmitt