Can rsync update files in place?

Eric Ziegast ziegast at vix.com
Wed May 8 17:53:05 EST 2002


Martin replied in an earlier thread:
> On 16 Apr 2002, Paul Slootman <paul+rsync at wurtel.net> wrote:
> > Is there a way to get rsync to not create a new file while transferring
> > and then rename it, but to instead update the existing file in place,
> > i.e. simply write those blocks that have been updated and leave the rest
> > alone?
> 
> That would be pretty hard, though not impossible.
> 
> I suspect it would be roughly as hard to put it into rsync as to 
> write a new program from scratch, depending on what features you need.
> 
> NetApp machines are kind of a special case because you can't (?)
> run an rsync program directly on the server.  So every block 
> is going across the network in at least one direction.
> 
> Perhaps Oracle's native replication features would be better?
> 
> Happy hacking,
> -- 
> Martin

NetApp snapshots are great for storing file deltas over time.  With
the introduction of their R100, they're enabling better practices for
doing disk to disk backup and recovery.  The only problem with the
current poduct and marketing is that the backup solution doesn't
address the storage waste of copying entire files instead of just
using incremental copies of the changed records.  Without incremental
record changes, multiple whole copies of the file would be stored in
snapshots instead of just the incremental changes.  (What a waste!)

The last time I looked, the block size on a NetApp WAFL filesystem was
4096 bytes. If you used the following algorithm, you could copy just
the changes:

  BEGIN
  {
     open source_file read_only
     open destination_file read_write
     for each block_position from 0 to int(source file size / 4096)
     {
        src_block = read source_file, 4096 bytes at block_position
        dst_block = read destination_file, 4096 bytes at block_position
        if src_block and dst_block are different
        {
           write src_block to destination_file at block_position
        }
     }
     if file is not block-aligned
     {
        src_block = read source_file, remaining bytes at block_position
        dst_block = read destination_file, remaining bytes at block_position
        if src_block and dst_block are different
        {
           write src_block to destination_file at block_position
        }
     }
     close source_file
     close destination_file
  }
  END

I've just written a Perl program that does an in-place incremental
block-level copy between files on a Unix system.

Take a look here:

   http://www.vix.com/~ziegast/src/blkcopy

Note: To protect myself, the program won't run properly until you
      modify the script to disable the annoying disclaimer statement.

If someone would like to try it out on a really large database
file (eg: MySQL, Oracle) and show the filesystem size differences
on a NetApp between using blkcopy versus other programs, let me
know.

If you work at Netapp or a backup software company, drop me a line
for more info.  You can modify your programs to support incremental
block-level backups just like the blkcopy script does.

--
Eric Ziegast




More information about the rsync mailing list