rsync 1tb+ each day
knnf6cy7w001 at sneakemail.com
Wed Feb 5 09:56:01 EST 2003
On Tue, 4 Feb 2003, Kenny Gorman kgorman-at-paypal.com |Rsync List| wrote:
> My question is this:
> Is it possible to tell rsync to update the blocks of the target file
> 'in-place' without creating the temp file (the 'dot file')?
It does not look like this is possible. In receiver.c around line 452 you
/* recv file data */
recv_ok = receive_data(f_in,buf,fd2,fname,file->length);
fd2 looks to always be a file descriptor generated via do_mkstemp.
> I can guarantee that no other operations are being performed on the
> file at the same time. The docs don't seem to indicate such an
Rsync works best over low bandwidth links for which the disk I/O is
minimal. You might try one of these ideas for your high-bandwidth
+ hack receiver.c so that receive_data uses fd1 (the original file)
- also comment out finish_transfer, which does the rename and
sets the permissions. If perms are important, then set them
- check the dest file closely to make sure it's not mangled
- This is completely untested. If you're not comfortable with
hacking and the usual followup debugging, then skip it.
+ compile and run rsync with profiling libraries to make sure that
it's slow where you think it's slow. I've been surprised
+ use a "dumber" file transfer method (FTP, netcat) that will be
faster on beefier hardware. Netcat is especially fast if you have
a private network where you don't have to worry about pesky things
For Oracle datafiles I've had excellent luck with a homebrew file transfer
that "compresses" blocks of zeros by sending a message that means "Hey!
I just read XXX blocks of nothing but zeroes." The receiver then creates
a sparse file on the destination for this area of zeroes. It's very handy
for making copies of the database for read-only purposes and it saves lots
of disk space. It doesn't help at all vs. something like netcat if your
datafiles are mostly full, tho.
PS: You can get info on netcat from:
More information about the rsync