rsync through a server storing the changes, time delayed rsync

[Matt McCutchen]

On Tue, 2006-02-21 at 11:28 -0500, Carson Gaspar wrote:
> --On Tuesday, February 21, 2006 4:48 PM +0100 Torbjörn Nordling 
> <tn at s3.kth.se> wrote:
> 
> > Problem:
> > I have two computers (one at work and one home) and I want to keep them
> > identical, but I cannot rsync them directly because when one is running
> > then the second is turned off. I also have access to 500 MB of storage
> > space on a server running continuously, which is not nearly enough to
> > hold all data stored on the two computers. So my idea is to use the
> > server for storing an rsync file list with checksums from the last update
> > and then upload the data that has changed to the server.
> >
> > Scenario:
> > Last thing I do at work is update the file list on the server and upload
> > all data the rsync identify as changed. Then at home I would connect to
> > the server and download the changed data, which is integrated by rsync in
> > correct files. Before I turn off my home computer then I would again
> > rsync the changes to the server. This would make the rsync delayed in
> > time.
> >
> > Question:
> > Has anyone tried something similar and could direct me towards how to
> > make it work? Or is this impossible today, but something that quite
> > simply could be built into rsync? Based on what I have read about how
> > rsync works it seems feasible to me.
> 
> What you're suggesting won't work. You have 2 options I can see:
> 
> 1)
> - Maintain a "last synced" timestamp on each client and the server.
> - To sync to the server, find all files with a mtime newer that the last 
> synced time, and copy them onto the server, then update the sync time on 
> that client and the server.
> - To sync from the server, check if its last sync time is different from 
> yours. If it is, copy the files and the timestamp from the server, then 
> delete the files from the server.

This arrangement doesn't give me a good feeling; it reminds me of an
unreliable two-way synchronization technique based on rsync that I
proposed a long time ago (remember that, Wayne? :)).  Potential issues
include other programs that modify mtimes and file deletion.  Torbjörn,
it sounds like you don't need a full two-way system but a
one-way-and-then-the-other-way system; those are much easier to do.

> 2)
> - Maintain 2 copies of the data on both clients (using snapshots would make 
> your life easier and storage requirements smaller)
> - Use rsync to generate batch change files between the old and new copies, 
> and copy them onto the server
> - download and apply pending batches on the other client

That would work nicely and reliably if you have the space.

Here's another technique you might consider if you have a high enough
inode quota on the server.  Get an rsync that supports the --times-only
quick-check behavior (e.g., apply just the times-only piece of the
experimental filter patch) and use it.  Keep the entire file structure
on the server with the correct attributes, but make all the files
zero-length.  To upload, just upload; if a file is changed on your
computer, its real data will overwrite the zero-length data on the
server, while unchanged files will match mtimes with zero-length files
on the server and their data won't be uploaded to take up server space.
To download, just download; the real and newer data on the server will
overwrite the data on your computer, while zero-length files on the
server will match mtimes with (and not overwrite) real files on your
computer.  Every so often, truncate the files on the server back to zero
length after you download.
-- 
Matt McCutchen
hashproduct at verizon.net
http://hashproduct.metaesthetics.net/