Using --link-dest works on small directory tree but not large one

[Bob Hutchinson]

On Thursday 06 Apr 2006 11:19, Robot Robot wrote:
> I'm using version 2.6.3 on OS X. I'm trying to use --link-dest to keep some
> rolling backups of production websites. There is about 2GB of data in
> 60,000 total files. I do an initial rsync to get an initial copy of the
> data, and each day thereafter I run rsync again with --link-dest to create
> a new daily backup. For some reason, in the subsequent runs, rsync updates
> a whole lot of files that haven't changed at all. The files in the source
> and in the link-dest location have identical contents, are the same size,
> and have the same modified date. I can't seem to find any patterns in the
> files it decides to bring back down the subsequent times. Sometimes it will
> bring down the entire directory tree again, and sometimes it will bring
> down slightly less than the whole tree.

Sounds like you want to try rsnapshot
http://rsnapshot.org/

>
> I've tried running the same commands on subsets of the directory tree, and
> everything works perfectly. The problem only appears when I try to sync my
> entire directory tree.
>
> This is what I'm doing:
>
> $ rsync -vv --progress --archive --compress --numeric-ids --delete -e ssh
> blah at blah.com:/usr/home/blah/websites/ /backup/websites/1
> $ rsync -vv --progress --archive --compress --numeric-ids --delete
> --link-dest=../1 -e ssh blah at blah.com:/usr/home/blah/websites/
> /backup/websites/2
> $ du -hsc /backup/websites/*
> 2.1G    /backup/websites/1
> 2.1G    /backup/websites/2
> 4.2G    total
>
> If, however, I run the following, it all seems to work perfectly:
>
> $ rsync -vv --progress --archive --compress --numeric-ids --delete -e ssh
> blah at blah.com:/usr/home/blah/websites/site1.com /backup/websites/1
> $ rsync -vv --progress --archive --compress --numeric-ids --delete
> --link-dest=../1 -e ssh blah at blah.com:/usr/home/blah/websites/site1.com
> /backup/websites/2
> $ du -hsc /backup/websites/*
> 35M    /backup/websites/1
> 0    /backup/websites/2
> 35M    total
>
> This is frustrating to debug because there's a lot of data being
> transferred, and it only occurs when I try to sync the full tree. Is this a
> bug? Am I doing something completely wrong and stupid? I'm happy to provide
> more detailed info if you need it. Thanks!!

-- 
-----------------
Bob Hutchinson
Midwales dot com
-----------------