Testing a transfer-only rsync tool

[Wayne Davison]

I found some time in the past week to work on a simple test app that
would hopefully help to answer a few questions that came up recently:

1. Can a single-process generator+receiver work well?  (Looks good so far,
   but I haven't run any multi-processor timing tests yet.)

2. How easy is it to use librsync?  (Pretty easy.)

3. How small would a transfer-only tool be?  (It's currently around 1400
   lines of C code, not counting the librsync code.  It was around 900
   lines when I first considered releasing a simple working version, but
   it keeps growing as I flesh out the more advanced features.)

4. Should rsync be separated into a scanning tool and a transfer tool?
   Or should it contain both bits but also allow the user to override
   the scanner to fully control what gets transferred?  Or should we
   just try to optimize the current protocol? (No answers yet, but I'm
   leaving toward the 2nd option above.)

My test tool takes in commands on stdin and outputs messages on stdout.
It forks a second process as specified via the commandline (which can be
any command that runs another rsync_xfer, either locally or remotely).
It then allows you to send AND/OR receive any files you specify (as well
as delete files, mkdir directories, etc.).  Keep in mind that this tool
does not attempt to do any of the "scan both systems, looking for files
that differ" task.

The code is still fairly young and while some of it is pretty good,
other bits show signs of being written in haste.  I've tested it on a
small number of scenarios so far, but nothing exhaustive.

Commands accepted by the tool on stdin (* means not yet tested):

cd REMOTE_DIR [LOCAL_DIR]                    chdir both sides at once
tmpdir REMOTE_PATH [LOCAL_PATH]              where temp-files go
get REMOTE_FILE [LOCAL_FILE [BASIS_FILE]]    rsync to the local system
put LOCAL_FILE [REMOTE_FILE [BASIS_FILE]]    rsync to the remote system
mvget REMOTE_FILE [LOCAL_FILE [BASIS_FILE]]  get, then delete REMOTE_FILE
mvput LOCAL_FILE [REMOTE_FILE [BASIS_FILE]]  put, then delete LOCAL_FILE
del FILE                                     delete a remote file
ldel FILE                                    delete a local file
md DIR                                       create a remote directory
lmd DIR                                      create a local directory
ln OLDNAME NEWNAME                           create a remote hard link
lln OLDNAME NEWNAME                          create a local hard link
sln OLDNAME NEWNAME                          create a remote symlink
lsln OLDNAME NEWNAME                         create a local symlink
mkdev NAME NUMBER                            create a remote device*
lmkdev NAME NUMBER                           create a local device*
quit                                         quit

Spaces in filenames need to be backslash-quoted, as do backslash
characters.  E.g. get This\ File.txt That\ File.txt

You run the program like this:

    rsync_xfer -vv ssh remote.com rsync_xfer -s

This starts up a local rsync_xfer process in double-verbose mode, and
tells it to run the "ssh remote.com rsync_xfer -s" command.  You can
make this latter command anything you like, as long as it starts up an
rsync_xfer with the slave (-s) option.

If you're feeling brave and you'd like to try it out, feel free, but
treat it like the pre-alpha code that it is.  Also keep in mind that
every time you tell the program to switch from get to put, all the
current outstanding get/put jobs must run to completion before any new
jobs start (which can slow down the transfer by reducing the pipe-lining
of data).

Some of the things yet to do:

- Need a way to override the per-file attributes on files we send (it
  currently preserves the attributes on each source file).
- Need a way to specify/set attributes for non-transferred files, new
  directories, and devices.
- It needs a way to output transfer statistics.
- There needs to be a timeout option.
- It needs some of the error-checking to be polished up.
- Some fatal errors might be better as warnings (if done right).
- There's no retry check if the file changes during the send.
- We need to catch SIGPIPE.
- There needs to be better handling of partially-transferred files.
- The code needs to be broken up into multiple files.
- There's no configuration support (it currently compiles on a modern
  Linux system).
- We might want an option that tells us to connect via socket to a
  particular hostname (instead of running a command).
- The code could use some more verbose-output messages.
- It needs more comments.

The code is here:

    http://www.clari.net/~wayne/rsync_xfer.c

You need to have librsync installed or available.  You compile the code
as you would expect:

    gcc -g -Wall -o rsync_xfer rsync_xfer.c -lrsync

..wayne..