--dry-run reliability ?

François daitheflu at free.fr
Sat Jun 14 18:35:08 MDT 2014

Hi Kevin, hi list,

First, thanks a lot for your answer. 

I'm not really sure I understand why a cp -al + rsync works where rsync alone 
doesn't, but I gave it a try and it seems it solved my issue. Well, mostly :

What I did :

1. cd /path/to/backups
2. cp -al ./backupWithOldACL ./backupWithNewACL
2. rsync -ahAXS --numeric-ids source /path/to/backups/backupWithNewACL
3. ln -s /path/to/backups/backupWithNewACL ./latest
4. Then I ran my script again (with --link-dest), which told me that it would 
need ~150MB to run successfully. Since I had ~30GB of free space, it sounded 
OK to me and went to sleep.
5. This morning, I saw in my log that the backup completed successfully. But I 
also saw that the destination device is full ! So the backup took a lot more 
than the 150MB that were supposed to be used.

So, I now think that I am misinterpreting the "Total transferred file size" 
line and that it probably shouldn't be used for what I'm trying to do. Can you 
confirm this ?

As a result, I'm thinking about changing my strategy to something like this :
1. Remove all the dry-run stuff,
2. Run rsync,
3. Detect the "device full" error,
4a. If detected, remove the oldest backup,
4b. If not, go to 6.
5. Go to 2.
6. Backup successful.

With your experience and knowledge, could you tell me what you think about 
this ? Is it a better idea than relying on a dry run ?

Thanks again,

Le vendredi 13 juin 2014, 17:20:11 Kevin Korb a écrit :
> The problem here isn't transfer size it is that rsync is storing the
> affected files with and without the ACL even though the file data is
> the same.
> The solution is to run a single manual rsync using the old method that
> predates --link-dest...
> 1, find your most recent non-ACL backup
> 2, cp -al thatbackup newbackup
> 3, rsync -ahAXS --numeric-ids your data over top of the newbackup dir
> When you do this rsync will set the ACLs on the existing files (and in
> every old backup containing the same inode) rather than duplicating
> the file to store it with and without the ACL.
> On 06/13/2014 05:14 PM, daitheflu at free.fr wrote:
> > Hi,
> > 
> > I'm using rsync to backup my files. I have also added a pre-backup
> > step where the script checks if there is enough space on the
> > destination for rsync to run successfully. The goal is to make room
> > (delete oldest backups) if that's not the case.
> > 
> > I'm running rsync with -ahAXS --numeric-ids --link-dest options.
> > 
> > For my pre-backup step, I'm currently using the same options +
> > --dry-run --no-h --stats. I then parse the "Total transferred file
> > size" line, add a 20% (!) padding and compare the result with the
> > result of a "df" call on destination.
> > 
> > It seemd to work properly until I decided to add some ACL on the
> > source device. I didn't really know how rsync would handle the
> > situation, but understood that it may have to backup all the files
> > (instead of hardlinking them) because of this change.
> > 
> > The dry run reported that it would need about 1400 MB and I have
> > around 30 000 MB of free space, so it should have ran properly.
> > Sadly, I got the "no space left on device" error.
> > 
> > So, is it safe to rely on the stats provided by a dry run to
> > achieve my goal ? Does the --dry-run option take care of
> > ACL/permissions changes ?
> > 
> > 
> > Thanks a lot for your work and help, rsync is really a wonderful
> > tool.
> > 
> > Cheers,
> --
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
> 	Kevin Korb			Phone:    (407) 252-6853
> 	Systems Administrator		Internet:
> 	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
> 	Orlando, Florida		kmk at sanitarium.net (personal)
> 	Web page:			http://www.sanitarium.net/
> 	PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~

More information about the rsync mailing list