rsync --link-dest option with the destination directory containing old files.

[Robert Bell]

Wayne,

Thanks for the prompt response.

The issue is around performance.  We have a backup area containing 4.5M
inodes in each backup, of which around 0.5M to 1M are directories.

Backing up to a new directory each time is far slower than backing up to
an almost completely filled directory.  Deleting a directory with 4.5M
inodes is non-trivial too.

(We are running on an SGI Altix 4700 with 128 cores, 500 Gbyte of
memory, with XFS file systems running the Data Migration Facility
product.  The target file systems for holding backups are on cheap and
slow SATA disc - we see at best 10,000 inode deletions per second, and
this can drop to fewer than 100 per second when the file systems are
busy with just a few housekeeping tasks.).

Starting with a new directory structure each time, so building 4.5M
inodes, and then destroying a similar directory and file structure which
is almost correct is very wasteful.

We do have a workaround, which keeps the hard-links numbers at nearly
their maximum level, but this involves a scan of the directory prior to
receiving each rsync and hence a performance penalty.  We would rather
avoid this.

It seems to us that there is a good case for the enhanced functionality.

"When a file is found in the destination which should be replaced by one
in the source, look in the --link-dest directory first for a candidate,
and hard-link that in preference to doing a copy from source to
destination."


Thanks

Regards
Rob. Bell              e-mail: Robert.Bell at csiro.au
--
Dr Robert C. Bell
CSIRO Advanced Scientific Computing, Technical Services Manager

Chief Technology Officer, Bureau of Meteorology / CSIRO
High Performance Computing and Communications Centre (HPCCC)

Street: HPCCC Level 11, 700 Collins Street, Docklands Vic 3008, Australia
Postal: HPCCC Level 11, GPO Box 1289, Melbourne Vic 3001, Australia
Phone +61 3 9669 8102, fax +61 3 9669 8112, mobile 0428 108 333, CSIRO 93 3810
http://www.hpsc.csiro.au/   http://www.hpccc.gov.au/



On Mon, 22 Dec 2008, Wayne Davison wrote:

> Date: Mon, 22 Dec 2008 20:56:09 -0800
> From: Wayne Davison <wayned at samba.org>
> To: Robert Bell <Robert.Bell at csiro.au>
> Cc: rsync at lists.samba.org
> Subject: Re: rsync --link-dest option with the destination directory
>     containing old files.
> Resent-Date: Tue, 23 Dec 2008 15:56:17 +1100 (EST)
> Resent-From: <Robert.Bell at csiro.au>
> 
> On Tue, Dec 23, 2008 at 10:22:42AM +1100, Robert Bell wrote:
>> we would like rsync to use a hard-link from ../backup.1 in
>> preference to a new copy from source/ to backup.0/ .
>
> That's not something that I plan to implement.  I'd suggest changing
> your naming to always use a new directory, which will allow you to
> remove the oldest backup directory at your leisure.
>
> ..wayne..
>