mirroring a huge file using rsync while it's being written to.

[Matt McCutchen]

On Tue, 2008-08-05 at 17:42 +0530, Yogesh Kumar wrote:
> In the application that I am building, I've to "incrementally" copy
> several files of large sizes (around 1 GB) from several machines (some
> 300 in number) spread across various geographic locations. We've been
> using scp until now and are exploring rsync for the same. In my
> application, these files are being written to (still being updated)
> while rsync will be reading from them. To clarify my point further, we
> invoke rsync to read a file from a remote server where it is being
> continuously written to by another daemon. So we want to rsync
> repeatedly every few minutes to get the latest appended information to
> the remote file.
> 
> I should like to know:
> 
> (1) While reading from a file, can rsync be made to wait for some
> duration after it has read the last byte in the file, to re-read the
> next installments of additions done to it by the write daemon on the
> remote server?

No.  You'll have to run rsync in a loop or from a cron job scheduled to
execute every few minutes.

> (2) When we know that the only additions/changes to the remote file
> will happen at its tail, i.e, all the "writes" to the file are
> basically appending to the file. 

Then use --append to avoid the work of rechecking the beginning of the
file.

> (3) While rsync is updating the local copy of the remote file (at the
> local end), and is merely appending to it all the time, can I read
> from it "live", ie, read it while it is being appended to. I am
> invoking rsync with the --inplace option. Will the data be
> inconsistent while I am reading from it?

Yes, you can read the destination file while rsync is appending to it.
The file will be incomplete but all the data in it so far will be
correct (as with wget).
      * 
Matt
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : http://lists.samba.org/archive/rsync/attachments/20080805/7b55f4ae/attachment.bin