disconnected synchronization (mostly unidirectional)

[Konrad Karl]

Hello,

On Wed, Mar 14, 2007 at 11:02:04AM -0500, Phil Howard wrote:
> On Sat, Mar 10, 2007 at 08:43:01PM +0100, Konrad Karl wrote:
> 

[ deleted ]

> | machine A (office) is, where most file changes/downloads etc happen.
> |                    and it has limited internet access (only proxy possible)
> |  
> | Machine B (home) has low bandwidth, is NATed etc.
> | 
> | So I would like to use an USB hard drive as transportation medium.

[ deleted]

> I have a somewhat different scenario, but one I think is sufficiently
> close that it could be adapted to yours.
> 
> I have the entire Gentoo mirror (around 50 GB) syncronized at home which
> is on low bandwidth (28.8K dialup).  I keep it in sync with rsync in the
> following way.  I take a list of all the files I do have at home (which
> can be carried over the USB device, though I send that from home to office
> ahead of time over the net).  I run rsync using the --exclude-from option
> giving it the name of that file.  It downloads files that are not in that
> list (new files and files I accidentally removed).  I then create a tar
> file from the downloaded directory and copy that tarball directly to the
> USB flash drive (no filesystem or mounting is involved).

I see but I want to get closer to a directly connected rsync with all its
benefits - looking at file length and attribute changes etc.

Right now I am really close using my hacked cpio which generates
sparse files (pls see my earlier post). I am still trying to optimize
the space requirements and speed and have played with a hacked
fuse-dbfs-0.6 (it does not store the file contents) but fuse-dbfs-0.6
becomes really slow if you have more than a few thousand files in one
directory (I have up to 25000 or so, unfortunately) - it implements
directories as a linear and unsorted list...

> 
> You may be able to do something similar by substituting your office files
> that need to be replicated at home for the remote mirror I used above.
> The possible scenario might then be (assuming Unix/BSD/Linux hosts):
> 
> 1.  Bring/get list of files already at home to office.
> 
> 2.  Use rsync to make replica of office files to a temporary area using
>     --exclude-from to limit to new files
> 
> 3.  Save the replica subset to USB flash drive.
> 
> 4.  At home, extract files from USB flash drive.
> 
> The big issue here is files that merely CHANGE are not detected.  To get
> better syncronization, dates in the list of files could be used to cross
> check dates of actual files.  Remove anything that has changed from the
> exclude list, and proceed as above.

using my sparse-file mirror rsync detects the changes quite nicely.

Thanks for your input,
Konrad