RSync with /really/ long file lists

[Jan-Benedict Glaw]

On Fri, 2004-12-03 19:09:11 -0800, Brandon Knitter <knitterb at blandsite.org>
wrote in message <20041203190911.sg5tkem6lgvko4gs at hurl.moil.blandsite.org>:
> 1) What strategies do other people employ?

Some time ago, I had a similar problem; basically, a little number of
application worked on some kind of master repository, that needed to be
synced to other locations (so it's a strict one-way sync).

Since rsync'ing was too slow for me (thus, the remote sites were
possibly too long running with inconsistent data, too long times between
two rsync runs, ...), I took another approach. I wrote a little lib
(loaded through LD_PRELOAD=... in front of the data-generating
applications) that intercepted open(), close(), read(), write(), ... and
replicated it over the network to the other sites.

This way, I only needed to stop the system once, replicate it, and
re-start the "master" side. However, this only works if you've got
*very* reliable hardware: a network outage etc. would block the master
servers.

Unfortunately, this code isn't available freely by any means, but the
idea is there. Maybe I'd rewrite it at some time, since it prooved to be
quite useful.

As an addition, maybe file-checking could be done in advance (size, time
stamps) so that stopping the whole system before accepting a remote site
could be avoided...

> 2) Is there a better tool out there for moving large amounts of files?

Not really. IFF you can actually use rsync, it's a really cool tool. I
use it eg. for keeping Debian repos in sync and it just works (TM).
Doing a more timely remote replication isn't rsync's task, so it isn't a
good fit for this.

> 3) Is there a way to save the filelist to a file instead of in memory?

Even if you'd press the file list to a file: you'd need to mmap() it, in
parts (4GB limit on 32bit hardware, which you aren't even allowed to
fully exploit...), which would complicate things, and make it possibly a
lot slower.

MfG, JBG
PS: Thanks to all who wrote code for rsync -- I'm really a happy user!

-- 
Jan-Benedict Glaw       jbglaw at lug-owl.de    . +49-172-7608481             _ O _
"Eine Freie Meinung in  einem Freien Kopf    | Gegen Zensur | Gegen Krieg  _ _ O
 fuer einen Freien Staat voll Freier Bürger" | im Internet! |   im Irak!   O O O
ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA));
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://lists.samba.org/archive/rsync/attachments/20041204/e864404f/attachment.bin