rsync is slowing down

[Wayne Davison]

You can implement such optimizations on top of rsync using either
excludes or the --files-from option.  For instance, if the sending
side maintained an exclude file of old directories that didn't need
to be transferred, you could write a script that would look for
updated items and remove the appropriate exclusion.  An exclude list
would have to be grabbed first from the remote side before it could
be used, though.

Using --files-from lets you use a remote list of items directly.
If you had a shell script like this:

#!/bin/sh
cd /starting/path || exit
touch /some/touchfile.new
for x in *; do
    case $x in
    [0-9][0-9][0-9][0-9])
	find $x -newer /some/touchfile ! -type d
	;;
    *)
	echo $x
	;;
    esac
done >/some/files-from.new
mv /some/files-from.new /some/files-from
mv /some/touchfile.new /some/touchfile

You could run an rsync command like this:

rsync -ar --files-from=:/some/files-from remotehost:/starting/path /to

The above script uses a separate touchfile from the files-from file
so that it won't miss files that got modified during the time that
the find is running but missed by this run.  It also avoids having
find report modified directories because rsync is being run with the
-r option.  You'd have to pre-create the /some/touchfile before the
first run.

You could, of course, construct your own script that implemented a
more complex update-check that was more like the one you suggested.
You might also need a more complex dir-scan algorithm, but that's
totally in your control.

..wayne..