disconnected synchronization (mostly unidirectional)

[Phil Howard]

On Sat, Mar 10, 2007 at 08:43:01PM +0100, Konrad Karl wrote:

| please consider the following scenario:
| (and pleast forgive if I have not googled enough but I was unsuccessful to
| find an app what exactly does what I want)
| 
| machine A (office) is, where most file changes/downloads etc happen.
|                    and it has limited internet access (only proxy possible)
|  
| Machine B (home) has low bandwidth, is NATed etc.
| 
| So I would like to use an USB hard drive as transportation medium.
| 
| First lets assume, A and B have been brought to sync somehow.
| 
| Afterwards I could do a disconnected rsync operation like follows:
| (assuming transfer A -> B)
| 
| 1.) get B's hard disk contents metadata  - perhaps the contents of
|    rsync's file list can be stored onto the transportation medium.
| 
| 2.) (in the office) let rsync generate its source file list from A
|     but instead of connecting to B and get the destinations file
|     list load B's list from the USB drive.
| 
| 3.) store the files which needed to be transfered and all the
|     other information (what has to be deleted, file attrs etc)
|     onto a shadow directory on the USB drive.
| 
| 4.) drive home :-)
| 
| 5.) again using rsync, copy the files to their final destination
|     and do all the other things required using the info stored
|     on the USB drive.
| 
| 6.) result: B should be equal to A 
| 
| For a first approach it would be fine to not use the
| "rsync" algorithm (transfer file differences only) because
| implementing this will perhaps require considerable work and
| hard drives are cheap nowadays.
| 
| What do you think?

I have a somewhat different scenario, but one I think is sufficiently
close that it could be adapted to yours.

I have the entire Gentoo mirror (around 50 GB) syncronized at home which
is on low bandwidth (28.8K dialup).  I keep it in sync with rsync in the
following way.  I take a list of all the files I do have at home (which
can be carried over the USB device, though I send that from home to office
ahead of time over the net).  I run rsync using the --exclude-from option
giving it the name of that file.  It downloads files that are not in that
list (new files and files I accidentally removed).  I then create a tar
file from the downloaded directory and copy that tarball directly to the
USB flash drive (no filesystem or mounting is involved).

You may be able to do something similar by substituting your office files
that need to be replicated at home for the remote mirror I used above.
The possible scenario might then be (assuming Unix/BSD/Linux hosts):

1.  Bring/get list of files already at home to office.

2.  Use rsync to make replica of office files to a temporary area using
    --exclude-from to limit to new files

3.  Save the replica subset to USB flash drive.

4.  At home, extract files from USB flash drive.

The big issue here is files that merely CHANGE are not detected.  To get
better syncronization, dates in the list of files could be used to cross
check dates of actual files.  Remove anything that has changed from the
exclude list, and proceed as above.