Steroids for Rsync!

[Danny Sauer]

On Wednesday 02 March 2005 06:11 pm, Scott Becker wrote:
> I've been researching the state of 'file alteration monitoring'
> technology on Linux. Famd uses dnotify to inefficently monitor a handful
> of directories. The replacement for dnotify is being worked on in the
> kenel and it's called inotify. If I understand it correctly and they get
> it finished, it would be an awesome addition to rsync. With it, you
> could run rsync to update a remote system (push mode) and keep it
> up2date. With inotify efficiently feeding 'file opened for write was
> just closed' notifications to rsync, it could efficiently and
> continuously mirror an active file system. With enough bandwidth, the
> time lag could be mere seconds.
>
> My challenge is to mirror 300 gigs of half meg files on to three remote
> filesystems, afap. Most of the files are 'write and leave for two
> months' with some being 'write 4 times a day' so rsync should be perfect
> if it can be told right after the file is created. We are installing
> fiber to our building so this might be the perfect combination to come
> up with a five second mirror performance.

[...]

Have you looked into the existing solutions for this problem, along the 
lines of network block devices used in a RAID-1?  I used that setup 
probably 5 years ago with a reliable network link, in order to keep 4 
machines in sync.  The ndb driver in Linux is much improved since then...  
Using the mdutils, I'm pretty sure that the software RAID susbsystem (yeah, 
I'm still assuming Linux) would handle disconnected remote drives, though 
you might have to manually bring them back into sync after network 
disconnections (probably with a script to monitor /proc/mdstat).

I agree that rsync is really cool, and that something like this would be 
handy, but I'm inclined to agree with Eli in that this might be better 
handled by having a daemon listen for changes and then send the changed 
file(s) up to the destination server(s), probably using the rsync protocol.  
This one-way sync thing might be even better handled by the RAID/nbd 
solution - since that's really what nbd was designed for. :)

--Danny