rsync 1tb+ each day

[Steve Bonds]

On Tue, 4 Feb 2003, Kenny Gorman kgorman-at-paypal.com |Rsync List| wrote:

> My question is this:
> 
> Is it possible to tell rsync to update the blocks of the target file
> 'in-place' without creating the temp file (the 'dot file')?

It does not look like this is possible.  In receiver.c around line 452 you
can see:

/* recv file data */
recv_ok = receive_data(f_in,buf,fd2,fname,file->length);

fd2 looks to always be a file descriptor generated via do_mkstemp.

> I can guarantee that no other operations are being performed on the
> file at the same time.  The docs don't seem to indicate such an
> option.

Rsync works best over low bandwidth links for which the disk I/O is
minimal.  You might try one of these ideas for your high-bandwidth
environment:
  + hack receiver.c so that receive_data uses fd1 (the original file)
    - also comment out finish_transfer, which does the rename and 
      sets the permissions.  If perms are important, then set them 
      manually.
    - check the dest file closely to make sure it's not mangled
    - This is completely untested.  If you're not comfortable with 
      hacking and the usual followup debugging, then skip it.
  + compile and run rsync with profiling libraries to make sure that 
    it's slow where you think it's slow.  I've been surprised 
    before.  ;-)
  + use a "dumber" file transfer method (FTP, netcat) that will be 
    faster on beefier hardware.  Netcat is especially fast if you have
    a private network where you don't have to worry about pesky things
    like authentication.

For Oracle datafiles I've had excellent luck with a homebrew file transfer
that "compresses" blocks of zeros by sending a message that means "Hey!  
I just read XXX blocks of nothing but zeroes."  The receiver then creates
a sparse file on the destination for this area of zeroes.  It's very handy
for making copies of the database for read-only purposes and it saves lots
of disk space.  It doesn't help at all vs. something like netcat if your
datafiles are mostly full, tho.

  -- Steve

PS:  You can get info on netcat from: 
     http://www.sans.org/rr/audit/netcat.php