--compare-dest weirdness

[Henri Shustak]

> Hi All,
> 
> I am writing a backup program for my computer. brief outline is as follows.
> 
> Running ubuntu 10.04
> 2 main partitions, / and /home, both ext3. 1 external usb hdd, ext3,
> mounted to /backups/main.
> once every couple of days, rsync backs up, using following command,
> everything worth backing up in / and /home partitions to a folder
> /backups/main/Full. command: "rsync -vrhRupElog --delete-during
> --delete-excluded --exclude-from=/backups/.config/full.exclude
> --log-file=/backups/.logs/full.rlog / /backups/main/Full/". this works
> perfectly
> 
> now what i want to do, is every few hours, rsync ONLY the files
> changed in the last few days to a folder in
> /backups/main/Incremental/YYYYMMddTHHmm, where the yyyy... is a string
> representing the date and time. (and then tar.gz the previous
> directory to save space) the command i am using for this is : "rsync
> -vrhRupElog --delete-during --delete-excluded
> --exclude-from=/backups/.config/incremental.exclude
> --log-file=/backups/.logs/incremental.rlog
> --compare-dest=/backups/main/Full/ /
> /backups/main/Incremental/Full/20100716T1355" the incremental.exclude
> differs from the full.exclude only in that i also exclude .mp3, .avi
> and other large and non-critical file types.
> 
> the problem is that rsync is behaving as though it is ignoring the
> compare-dest option. it throws no error about this dir not existing,
> and i have tried giving it as a relative dir to the destination with
> same result.
> 
> interestingly, i tried to see if something was wrong with my statments by doing:
> mkdir ~/rsynctest/dir1
> mkdir ~/rsynctest/dir2
> mkdir ~/rsynctest/dir3
> 
> nano ~/rsynctest/dir1/file1 (wrote the line hello world and saved)
> nano ~/rsynctest/dir1/file2 (wrote the line hello and saved)
> 
> cp ~/rsynctest/dir1/file* ~/rsynctest/dir2/
> 
> checked md5sums of both files in both dirs to ensure they were identical
> 
> the did rsync -rvu --compare-dest=/home/kevin/rsynctest/dir2/
> /home/kevin/rsynctest/dir1/ /home/kevin/rsynctest/dir3/
> 
> the two files from dir1 were copied to dir3, even though identical
> copies, with the same names, existed in dir2.
> 
> i also tried this again with the compare dir relative to the dest dir,
> i.e.: rsync -rvu --compare-dest=../dir2/ /home/kevin/rsynctest/dir1/
> /home/kevin/rsynctest/dir3/ again with same result
> 
> is this a bug, or have i got the wrong end of the stick regarding --compare-dir?
> 

Hopefully, someone on the list will be able to provide you with some further details regarding what is happening. 

In the mean time I suggest that you consider looking at rsnapshot, rdiffbackup, link-backup or LBackup.

All of the projects listed above and many more are listed at the following URL : http://www.lbackup.org/alternatives

Certainly, with LBackup (disclaimer - I am a developer on this project) you are able to configure the backup to have just one or two hard linked rotations by specifying the following in the backup configuration file : 

numRotations=1 # only the most recent successful backup will be kept
numRotations=2 # only the two most recent successful backups will be kept

LBackup will save space even if you save 50, 500 or 1000 backups due to hard linking files which have not been changed or moved. The one issue is that it is important not to exceed the your file systems inode limits.

With regards the creation of your compressed tar you could add a post action into the resources/post-actions directory of your configuration directory and create a compressed tar of the most recent backup. If you wanted to you could even rsync this backup to one or more remote systems. The post action sub system of each LBackup configuration directory makes this kind of customized setup trivial. For example there is post action script bundled with LBackup which will generate these dated linked directories within your backup set. The example script is called : BACKUP-300-Generate-Time-Based-Backup-Links-Directory.bash

This script makes altering the format of the links, having multiple  formats or different formats for different a snap.

If you are quite sure you want to roll your own rsync based backup then that is fantastic.

The only suggestions I have with regards why your current system is not working is that perhaps trying the --link-dest rather than the --compare-dest option may provide some further insight into what is not working. However, I see no reason why one would work while the other would not.

I also suggest that you try to check why the files are being updated and not matched by adding the --itemize-changes option. This option should provide some further details. In addition, you may want to also add some additional verbosity by adding -vvv as an option. You currently have just one -v.

Are you running rsync as root or as a non-privilged user. I assume you are running as root because you are using the --owner and --group options. Would you confirm that this is the case.

Also I wonder if the --compare-dest option would work if you removed the --relative option? It could be that option is causing some sort of problem with regards the --compare-dest option?

Finally, ensure that permissions and time stamps are being correctly preserved on destination. If the permissions are not being preserved, then this may be causing the problems with --compare-dest which you have outlined. 

I hope this helps and if not then hopefully someone else will chime in to provide you with some further ideas and tips or maybe spot something which I have missed.


---------------------------------------------------------------------
This email is protected by LBackup, an open source backup solution. 
Free as in freedom; LBackup is licensed under the GNU GPL
http://www.lbackup.org