rsync to iSCSI over WAN

[Benjamin R. Haskell]

On Fri, 23 Jul 2010, Tom Christensen wrote:

> I am running rsync in cygwin on windows. I am attempting to backup a 
> somewhat large data store (750GB) to a remote site. As its windows and 
> preserving permissions exactly is important, I have an iSCSI drive 
> mounted on the local system across a somewhat slow WAN link (IE, it 
> would take about 3 months to copy the datastore over it).  
> Unfortunately, since this appears as a "local" copy to rsync, it 
> always copies whole files. Even though it is a "local" copy, I want to 
> only send diffs, as we have large files that have small changes daily. 
> Reading the man page, and everything I can find on the net I don't see 
> an option to force diffs only/rsync protocol, is this possible?

I didn't see a "don't use --whole-file"-type option.  But the following 
doesn't use whole-file copies, as alluded to in the --whole-file 
documentation ("..., but only if no batch-writing option is in 
effect."):

Create a file with the batched updates:
rsync -a --only-write-batch=batched-updates local/disk/ iscsi/target/

Replay those updates:
rsync -a --read-batch=batched-updates iscsi/target/
(or just:)
./batched-updates.sh

Below is a test script that demonstrates.

-- 
Best,
Ben

Steps:
1. Set up a test directory
2. Create a test file of 100MB of 0's, 12345, 100MB of 0's.
3. Sync it so there are two identical files.
4. Change the 12345 in the test file to 54321.
5. Generate the batch script.
6. Sync using the file.
7. Clean up.

#!/bin/sh
mkdir -p /tmp/rsync-test
cd /tmp/rsync-test
mkdir src
perl -we 'print "0" x 100e6, 12345, "0" x 100e6' > src/file
rsync -a src/ dest/
perl -we 'print "0" x 100e6, 54321, "0" x 100e6' > src/file
rsync -a --only-write-batch=batched-updates src/ dest/
md5sum src/file dest/file
ls -l batched-updates*
./batched-updates.sh
md5sum src/file dest/file
rm -rf /tmp/rsync-test