rsync - using a --files-from list to cut out scanning. How to handle deletions?

[Robert Bell]

Folks,

We use rsync extensively for protecting data by making backups. 
Thank you to the authors and maintainers.


Like many others, we use the --link-dest option to cut down on the space
occupied by the backups.

Unlike many others, we re-cycle old backup directories.  Since most file
systems change only slowly (ours average about 0.5% of files and about
1.5% of data being churned each day), a recycled directory is a good
start for the next backup.  Our most common case is that a directory
from 5 days ago becomes the target for the current backup, with the
yesterday's backup being provided by a --link-dest= setting.

Since the source file system changes only slowly, I have been thinking
about ways to speed up the backups in the future.  One way is to have
the backups deal only with files that have changed on the source since
the last backup.  This would save having to scan the whole source and
destination areas each time a backup is done.  The Linux inotify
capability looks like it might be useful for collecting a list of
changed files.

Has anyone done this?

However, there is one case that I have not been able to get to work in a
test of rsync.  This is the case where a file exists in the destination,
does not exist in the source, but is named in the --files-from= list.
This would be the case if a file had been deleted from the source.  We
would want rsync in this case to delete the file on the destination.

However, with a test command like:

rsync -a -i --delete --files-from=list --link-dest=../linked source/ dest

I was unable to get rsync to delete on the destination a file which did
not exist in the source but was named in the list.  rsync baulked at a
file being listed that was not in the source.  For example:

rsync: link_stat "/data/flush/inter/bel107.80527/source/0yyy" failed: No such file or directory (2)

[The test file 0yyy existed in the destination, the link-dest area and in 
the list, but not in the source.]

Thanks to those who have read down to here.  :-)

Regards
Rob. Bell              e-mail: Robert.Bell at csiro.au
--
Dr Robert C. Bell, BSc (Hons) PhD
Technical Services Manager
Advanced Scientific Computing
CSIRO IM&T

Phone: +61 3 9669 8102 | Mobile: +61 428 108 333 | CSIRO 93 3810
Robert.Bell at csiro.au | http://www.csiro.au/ | http://www.hpsc.csiro.au/
Addresses:
Street: CSIRO ASC Level 11, 700 Collins Street, Docklands Vic 3008, Australia
Postal: CSIRO ASC Level 11, GPO Box 1289, Melbourne Vic 3001, Australia

PLEASE NOTE

The information contained in this email may be confidential or
privileged. Any unauthorised use or disclosure is prohibited. If you
have received this email in error, please delete it immediately and
notify the sender by return email. Thank you. To the extent permitted
by law, CSIRO does not represent, warrant and/or guarantee that the
integrity of this communication has been maintained or that the
communication is free of errors, virus, interception or interference.

Please consider the environment before printing this email.