rsync for realtime filesystem replication.
Michael Welles
mike at bangstate.com
Tue Nov 6 08:01:28 EST 2001
Apologies in advance for a kind of long question:
I've written an application called changedfiles
(http://www.bangstate.com/software.html#changedfiles), which consists of two
parts: a linux kernel module that reports all file operations to a device
special, and a userspace daemon which can take arbitray shell action on files
matching regexp rules specified in a config file.
E.G., to configure a "hot folder" that would automatically convert .jpg's to
.pngs, I'd define a rule like follows:
RULE ^/export/home/image-converter.*\.jpg$
OPENW /usr/bin/convert _FILE_ `dirname _FILE_`/`basename _FILE_ .jpg`.png
I can use it currently for sorta "hacked' realtime replication with rsync, by
defining a command line rsync as the action for a given rule. However, this
is inefficient, as it leads to repeated exec's for file actions ( e.g., the
extraction of a tar file results in N shell executions, one for each
filesystm operation: the mkdir for the directory, the open for write for
each file, etc...
I'm interested in integrating rsync into the userspace daemon itself. I
looked briefly at librsync, which is a part of rproxy, but at a very cursory
glance it didn't look like it was what I was looking for (HTTP as the
transport, for example). I'd like to be able to have the changedfiles
daemon talk to the rsyncd on the remote server directly.
Is this feasable at all? Does anyone have any pointers as to where I could
go for information on integrating rsync into a C or C++ app? Am I just
simply nuts? How much pain am I setting myself up for?
As an example of why I'm interested, this is the ugly rule I can use
currently for half-assed realtime replication:
RULE ^/home/mike/test
RETRY 5
TIMEOUT 300
IGNORE \.Xauthority
OPENW rsync --rsh=ssh -a _FILE_ faustus:/mirrors/otto_FILE_
MKDIR rsync --rsh=ssh -a _FILE_ faustus:/mirrors/otto_FILE_
SYMLINK rsync --rsh=ssh -a _NEW_FILE_ faustus:/mirrors/otto_NEW_FILE_
UNLINK ssh faustus rm -f /mirrors/otto_FILE_
RMDIR ssh faustus rmdir /mirrors/otto_FILE_
RENAME ssh faustus "rm /mirrors/otto/_OLD_FILE_"; rsync --rsh=ssh -a
_NEW_FILE_ faustus:/mirrors/otto/_NEW_FILE_
More information about the rsync
mailing list