rsync for realtime filesystem replication.

Michael Welles mike at bangstate.com
Tue Nov 6 08:01:28 EST 2001


Apologies in advance for a kind of long question: 

I've written an application called changedfiles 
(http://www.bangstate.com/software.html#changedfiles), which consists of two 
parts:   a linux kernel module that reports all file operations to a device 
special, and a userspace daemon which can take arbitray shell action on files 
matching regexp rules specified in a config file.    

E.G., to configure a "hot folder" that would automatically convert .jpg's to 
.pngs,  I'd define a rule like follows: 

RULE ^/export/home/image-converter.*\.jpg$
OPENW  /usr/bin/convert _FILE_ `dirname _FILE_`/`basename _FILE_ .jpg`.png

I can use it currently for sorta "hacked' realtime replication with rsync, by 
defining a command line rsync as the action for a given rule.  However, this 
is inefficient, as it leads to repeated exec's for file actions ( e.g., the 
extraction of  a tar file results in N shell executions, one for each 
filesystm operation:   the mkdir for the directory, the open for write for 
each file, etc...

I'm interested in integrating rsync into the userspace daemon itself.  I 
looked briefly at librsync, which is a part of rproxy, but at a very cursory 
glance it didn't look like it was what I was looking for (HTTP as the 
transport, for example).    I'd like to be able to have the changedfiles 
daemon talk to the rsyncd on the remote server directly. 

Is this feasable at all?  Does anyone have any pointers as to where I could 
go for information on integrating rsync into a C or C++ app?  Am I just 
simply nuts?  How much pain am I setting myself up for? 
 
As an example of why I'm interested, this is the ugly rule I can use 
currently for half-assed realtime replication: 

RULE ^/home/mike/test
RETRY 5
TIMEOUT 300
IGNORE \.Xauthority
OPENW   rsync --rsh=ssh -a _FILE_  faustus:/mirrors/otto_FILE_
MKDIR   rsync --rsh=ssh -a _FILE_  faustus:/mirrors/otto_FILE_
SYMLINK rsync --rsh=ssh -a _NEW_FILE_  faustus:/mirrors/otto_NEW_FILE_
UNLINK  ssh faustus rm -f /mirrors/otto_FILE_
RMDIR   ssh faustus rmdir /mirrors/otto_FILE_
RENAME  ssh faustus "rm /mirrors/otto/_OLD_FILE_"; rsync --rsh=ssh -a 
_NEW_FILE_ faustus:/mirrors/otto/_NEW_FILE_




More information about the rsync mailing list