rsync for realtime filesystem replication.

Dave Dykstra dwd at bell-labs.com
Wed Nov 7 03:57:46 EST 2001


Unfortunately, it's not practical to turn rsync into a library or
reimplement the entire rsync protocol.  Many people have wanted this, but
the answer is always the same.  The protocol used between the two halves of
an rsync connection is highly optimized and specific to the rsync
application; it isn't at all intended to be anything that can be
standardized or implemented independently.  There are even many places in
the code where it checks the version number of the protocol supported by
the remote end and inserts extra bytes here and there; it's quite a mess
that way.

Probably the best you can do is come up with a new implementation, most
likely based on librsync, that implements both halves and copies a single
file using the rsync rolling checksum algorithm.

- Dave Dykstra


On Mon, Nov 05, 2001 at 04:01:28PM -0500, Michael Welles wrote:
> Apologies in advance for a kind of long question: 
> 
> I've written an application called changedfiles 
> (http://www.bangstate.com/software.html#changedfiles), which consists of two 
> parts:   a linux kernel module that reports all file operations to a device 
> special, and a userspace daemon which can take arbitray shell action on files 
> matching regexp rules specified in a config file.    
> 
> E.G., to configure a "hot folder" that would automatically convert .jpg's to 
> .pngs,  I'd define a rule like follows: 
> 
> RULE ^/export/home/image-converter.*\.jpg$
> OPENW  /usr/bin/convert _FILE_ `dirname _FILE_`/`basename _FILE_ .jpg`.png
> 
> I can use it currently for sorta "hacked' realtime replication with rsync, by 
> defining a command line rsync as the action for a given rule.  However, this 
> is inefficient, as it leads to repeated exec's for file actions ( e.g., the 
> extraction of  a tar file results in N shell executions, one for each 
> filesystm operation:   the mkdir for the directory, the open for write for 
> each file, etc...
> 
> I'm interested in integrating rsync into the userspace daemon itself.  I 
> looked briefly at librsync, which is a part of rproxy, but at a very cursory 
> glance it didn't look like it was what I was looking for (HTTP as the 
> transport, for example).    I'd like to be able to have the changedfiles 
> daemon talk to the rsyncd on the remote server directly. 
> 
> Is this feasable at all?  Does anyone have any pointers as to where I could 
> go for information on integrating rsync into a C or C++ app?  Am I just 
> simply nuts?  How much pain am I setting myself up for? 
>  
> As an example of why I'm interested, this is the ugly rule I can use 
> currently for half-assed realtime replication: 
> 
> RULE ^/home/mike/test
> RETRY 5
> TIMEOUT 300
> IGNORE \.Xauthority
> OPENW   rsync --rsh=ssh -a _FILE_  faustus:/mirrors/otto_FILE_
> MKDIR   rsync --rsh=ssh -a _FILE_  faustus:/mirrors/otto_FILE_
> SYMLINK rsync --rsh=ssh -a _NEW_FILE_  faustus:/mirrors/otto_NEW_FILE_
> UNLINK  ssh faustus rm -f /mirrors/otto_FILE_
> RMDIR   ssh faustus rmdir /mirrors/otto_FILE_
> RENAME  ssh faustus "rm /mirrors/otto/_OLD_FILE_"; rsync --rsh=ssh -a 
> _NEW_FILE_ faustus:/mirrors/otto/_NEW_FILE_




More information about the rsync mailing list