Backup scripts - recycling old backup directories

Robert Bell Robert.Bell at
Thu Sep 11 22:31:47 MDT 2014


Kevin Korb wrote:
> Have you considered more advanced methods such as subvolume snapshots
> provided by ZFS and BTRFS?  At work we were forced to abandon rsync
> - --link-dest because of the amount of time it takes to delete old
> backups when the data is primarily many millions of small files
> (shared web hosting company).

We don't have more advanced methods like subvolume snapshots available
to us.

However, we can recycle backup directories.

When we started using rsync with --link-dest back in about 2007, we
deleted old backups, but realised soon after that we could recycle old

With daily backups, we find about 1.5% of the data and 0.5% of the files
change from one day to the next, so a directory from about 5 days ago
will typically be only 5-10% wrong and can be recycled to be the target
of the latest directory - that's a lot better than recreating the whole
directory tree for a new backup, and then deleting a whole old directory

We use --delete of course.

Hope this helps someone.


Dr Robert C. Bell
HPC National Partnerships | Scientific Computing
Information Management and Technology
T +61 3 9669 8102 Alt +61 3 8601 3810 Mob +61 428 108 333
Robert.Bell at<mailto:Robert.Bell at> | |
Street: CSIRO ASC Level 11, 700 Collins Street, Docklands Vic 3008, Australia
Postal: CSIRO ASC Level 11, GPO Box 1289, Melbourne Vic 3001, Australia

The information contained in this email may be confidential or privileged.
Any unauthorised use or disclosure is prohibited.  If you have received
this email in error, please delete it immediately and notify the sender by
return email. Thank you.  To the extent permitted by law, CSIRO does not
represent, warrant and/or guarantee that the integrity of this
communication has been maintained or that the communication is free of
errors, virus, interception or interference.

Please consider the environment before printing this email.

More information about the rsync mailing list