disconnected synchronization (mostly unidirectional)
kk_konrad at gmx.at
Wed Mar 14 18:58:43 GMT 2007
On Wed, Mar 14, 2007 at 11:02:04AM -0500, Phil Howard wrote:
> On Sat, Mar 10, 2007 at 08:43:01PM +0100, Konrad Karl wrote:
[ deleted ]
> | machine A (office) is, where most file changes/downloads etc happen.
> | and it has limited internet access (only proxy possible)
> | Machine B (home) has low bandwidth, is NATed etc.
> | So I would like to use an USB hard drive as transportation medium.
> I have a somewhat different scenario, but one I think is sufficiently
> close that it could be adapted to yours.
> I have the entire Gentoo mirror (around 50 GB) syncronized at home which
> is on low bandwidth (28.8K dialup). I keep it in sync with rsync in the
> following way. I take a list of all the files I do have at home (which
> can be carried over the USB device, though I send that from home to office
> ahead of time over the net). I run rsync using the --exclude-from option
> giving it the name of that file. It downloads files that are not in that
> list (new files and files I accidentally removed). I then create a tar
> file from the downloaded directory and copy that tarball directly to the
> USB flash drive (no filesystem or mounting is involved).
I see but I want to get closer to a directly connected rsync with all its
benefits - looking at file length and attribute changes etc.
Right now I am really close using my hacked cpio which generates
sparse files (pls see my earlier post). I am still trying to optimize
the space requirements and speed and have played with a hacked
fuse-dbfs-0.6 (it does not store the file contents) but fuse-dbfs-0.6
becomes really slow if you have more than a few thousand files in one
directory (I have up to 25000 or so, unfortunately) - it implements
directories as a linear and unsorted list...
> You may be able to do something similar by substituting your office files
> that need to be replicated at home for the remote mirror I used above.
> The possible scenario might then be (assuming Unix/BSD/Linux hosts):
> 1. Bring/get list of files already at home to office.
> 2. Use rsync to make replica of office files to a temporary area using
> --exclude-from to limit to new files
> 3. Save the replica subset to USB flash drive.
> 4. At home, extract files from USB flash drive.
> The big issue here is files that merely CHANGE are not detected. To get
> better syncronization, dates in the list of files could be used to cross
> check dates of actual files. Remove anything that has changed from the
> exclude list, and proceed as above.
using my sparse-file mirror rsync detects the changes quite nicely.
Thanks for your input,
More information about the rsync