rsync --link-dest option with the destination directory containing old files.

[Robert Bell]

Way back:
> > ---------- Forwarded message ---------- Date: Mon, 2 Jun 2003
> > 09:56:50 -0700 (PDT) From: Michael Rubel <mrubel at
> > galcit.caltech.edu> To: rsync at lists.samba.org Subject:
> > --link-dest when target and compare_dir both have file
> > 
> > Hi J.W. et al,
> > 
> > Kevin Everets was kind enough to inform me about some strange
> > behavior in his backup script, which seems to be the result of
> > --link-dest behaving unexpectedly in the case where target/ is
> > already populated with older versions of the same file.
> > 
> > Here's the situation:
> > 
> > We want to do: $ rsync -a --link-dest=../backup.1 source/ backup.0/
> > 
> > There is a file present under all three directories.  Suppose that
> > it the version in backup.1/ is identical to the version in source/,
> > and that backup.0/ contains an older version.
> > 
> > In this case, I would expect: 1. the version in backup.0 should be
> > unlinked 2. a new hard link should be created in backup.0/ to the
> > copy in backup.1/
> > 
> > In fact, rsync (at least as of 2.5.6) seems to copy the full file:
> 
> --link-dest is built on --compare-dest and behaves like --compare-dest
> as indicated in the manpage.
> 
> Observe the description of --compare-dest (ALLCAPS added for
> emphasis): This  option instructs rsync to use DIR on the des­
> tination machine as an additional directory to com­ pare destination
> files against when doing transfers IF THE FILES ARE MISSING IN THE
> DESTINATION DIREC­ TORY.
> 
> In other words, the files in the --link-dest location will only be
> used if there is no existing file in the destination.  The best way to
> use --link-dest is to have an empty destination.
> 
> For a link-dest based rotating backup that reuses directory names the
> best bet is to do a "rm -r $dest; mkdir $dest" or to "rsync -a
> --link-dest=../backup.1 $source temp; mv temp backup.0"
> 
> -- ________________________________________________________________
> J.W. Schultz            Pegasystems Technologies email address:
> jw at pegasys.ws
> 
> 
> 
rsync folks,

Can this behaviour be circumvented, or can an option be provided to
change this behaviour?

When doing precisely the type of backups listed earlier in the post:

   rsync -a --link-dest=../backup.1 source/ backup.0/

then we would like rsync to use a hard-link from ../backup.1 in
preference to a new copy from source/ to backup.0/ .

In particular, the removal suggested can be a very slow operation - we
have a backup area containing about 4.5M inodes.

Currently, a removal is the only way we have to stop the hard-links
from leaking when we re-use directories.



Regards
Rob. Bell              e-mail: Robert.Bell at csiro.au
--
Dr Robert C. Bell
CSIRO Advanced Scientific Computing, Technical Services Manager

Chief Technology Officer, Bureau of Meteorology / CSIRO
High Performance Computing and Communications Centre (HPCCC)

Street: HPCCC Level 11, 700 Collins Street, Docklands Vic 3008, Australia
Postal: HPCCC Level 11, GPO Box 1289, Melbourne Vic 3001, Australia
Phone +61 3 9669 8102, fax +61 3 9669 8112, mobile 0428 108 333, CSIRO 93 3810
http://www.hpsc.csiro.au/   http://www.hpccc.gov.au/

---------- Forwarded message ----------
Date: Mon, 22 Dec 2008 12:00:32 +0000
From: rsync-request at lists.samba.org
Reply-To: rsync at lists.samba.org
To: rsync at lists.samba.org
Subject: rsync Digest, Vol 72, Issue 20
Resent-Date: Mon, 22 Dec 2008 23:00:48 +1100 (EST)
Resent-From: <Robert.Bell at csiro.au>

Send rsync mailing list submissions to
 	rsync at lists.samba.org

To subscribe or unsubscribe via the World Wide Web, visit
 	https://lists.samba.org/mailman/listinfo/rsync
or, via email, send a message with subject or body 'help' to
 	rsync-request at lists.samba.org

You can reach the person managing the list at
 	rsync-owner at lists.samba.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of rsync digest..."