Feature I'd love to see: --move-dest

[Thomas Gutzler]

Hi,

This seems to be a reappearing request. Interestingly enough, I wanted
the exact same thing a while ago and nobody was able to suggest a quick
solution without implementing a new feature. I didn't have the time to
read into the rsync source code, so I decided to write a little perl
script, which works similar to rsnapshot. It
- keeps one exact copy of the source which is updated every time
- stores the changes between the current and the last run in an extra
directory which is labelled interval_date_time
- supports different intervals by merging several more frequent ones
into one less frequent (7 daily -> 1 weekly)
- does not use hard links

At the moment, I run it about 5 times a day on a 3TB file system (1.2M
files) with about 400MB changes each time and it takes around half an hour.

I haven't put an amazing amount of work into it because it was never
supposed to be used by anyone else than myself; it's not foolproof but I
am willing to share it around and improve it if there is interest.

Cheers,
  Tom

Josh Hanson wrote:
> David,
> 
> You make some good points. However, I still think a --move-dest
> feature would be handy.
> 
> First of all, --link-dest is nice on platforms and file systems that
> support hard links, but not all platforms and file systems do, and
> some that do have tagged them on as an afterthought but don't support
> them very well, or very efficiently.
> 
> Second, even on systems that do support it, for frequent backups of
> large numbers of files which may change only rarely, the size of all
> the hard links may not be negligible.
> 
> Finally, it's always nice, given the option, to avoid hard links and
> keep a one-to-one relationship between the "files" I see (i.e. the
> links) and the actual allocated regions on the disk. Hard links make
> it impossible to know just how much space each backup takes up. If my
> backup disk is filling up, I'd like to be able to see which files are
> taking the most space; I might decide to manually delete older copies
> of large files, and I won't want to go through and delete every
> single link to the file.
> 
> -- Josh
> 
> 
> 
> -----Original Message----- From: overton.d at googlemail.com
> [mailto:overton.d at googlemail.com] On Behalf Of David Overton Sent:
> Monday, December 08, 2008 5:35 PM To: Josh Hanson Cc:
> rsync at lists.samba.org Subject: Re: Feature I'd love to see:
> --move-dest
> 
> Hi,
> 
> I'm curious as to why you need a new option rather than just using 
> --link-dest.  Both --link-dest and --copy-dest already allow the "two
>  very common restor scenarios" that you mention and --link-dest uses 
> hardly any extra disk space than your --move-dest suggestion.  You 
> also get the added advantage that each previous backup continues to 
> look like a full "snapshot".
> 
> David
> 
> 2008/12/9 Josh Hanson <JoshHanson at capriomgt.com>:
>> Looking over rsync's --compare-dest, --copy-dest, and --link-dest 
>> options, there's one thing I really wish I could do, and I've been 
>> looking for a few years now for a program that could do it: Instead
>> of copying or hard-linking from the compare directory to the target
>>  directory, I'd like to move the files.
>> 
>> That is, whenever the file already exists in the compare directory,
>> it is moved forward into the target directory; but if the existing
>> file is different or doesn't exist, then a new file is made in the
>> target directory.
>> 
>> The result would be that, after several backups (each to a new 
>> directory, perhaps named with the current date) the most recent 
>> directory would be a snapshot of the source, and each previous
>> directory would have previous versions of files or copies of
>> deleted files. (This is effectively the opposite result of
>> --compare-dest, which gives you a snapshot of the first version of
>> the files, with each successive directory storing only the changed
>> files.) This makes two very common restore scenarios very simple:
>> restoring the entire directory tree to the most recent version
>> after a drive failure, and restoring a previous version of a single
>> file after it was accidentally over-written, even if the modified
>> version has since been backed up.
>> 
>> I looked over the rsync source to see if I could add this
>> functionality myself, but I'm really not a very good programmer and
>> it's definitely beyond me. To someone familiar with the source,
>> however, I have to imagine it would be a fairly straightforward
>> feature to add... just add replace the copy operation in
>> --copy-dest or the link operation in --link-dest with a move
>> operation.
>> 
>> -- Josh -- Please use reply-all for most replies to avoid omitting
>> the mailing list. To unsubscribe or change options:
>> https://lists.samba.org/mailman/listinfo/rsync Before posting,
>> read: http://www.catb.org/~esr/faqs/smart-questions.html
>> 

-- 
Thomas Gutzler
Optical+Biomedical Engineering Laboratory
The University of Western Australia
http://obel.ee.uwa.edu.au/
phone: +61 8 6488 3916