Rsync help needed...

[Linus Hicks]

Matt McCutchen wrote:
> On Fri, 2006-02-24 at 11:08 -0800, lsk wrote:
> 
>>////****/// lsk:- Thanks for the clarification Wayne, in my case no one
>>would be allowed to use the destination file until the process is
>>complete. As soon as my destination server is upgraded to the newer
>>version of rsync which supports --inplace option I am going to try with
>>in-place and -no-whole-file option without -- checksum since rsync
>>algorithm does it.
> 
> 
> --no-whole-file (i.e. incremental transfer) is the default except for
> local transfers, during which disk I/O is more of a limiting factor than
> network I/O; you need not specify it explicitly.
> 

Matt,
Some comments and questions...

I did something similar to what lsk is doing a few months back, I believe using 
rsync 2.6.5. I wrote a script to query the database for all the datafiles and 
rsync'ed them individually by specifying the full path to the file. What I found 
was that if I didn't use --no-whole-file, it did operate in whole-file mode. I 
was not doing local transfers, so is there some other condition that causes it 
to default to whole-file mode?

The issue of not using --inplace and atomically moving it over the original is 
complicated by using --temp-dir. lsk has not raised the issue of not having 
enough room for a second copy of any of his datafiles, so he probably isn't 
using --temp-dir. However, the statement you made earlier in this thread (quoted 
below) needs to be extended to account for the case where a --temp-dir resides 
on a different partition:

"Not exactly: if --inplace is not used, rsync will write a temporary file
and atomically move it over the original.  --inplace uses less disk
space but does not provide atomicity and, according to the man page,
reduces the efficiency of the incremental transfer algorithm."

And a performance question: would it be faster to pass the complete list of 
datafiles to rsync in one fell swoop, for instance using --files-from rather 
than running rsync individually on each one?

Linus