--dry-run reliability ?

[Kevin Korb]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

The difference is pretty simple.  An ACL (or permissions or ownership)
change does not require the transfer of a file only a change to the
file's attributes.  Therefore it does not affect the transfer size.

When you use the cp -al then rsync method rsync doesn't pay any
attention to previous backups therefor it just changes the ACL which
affects all links to the same inode.

Rsync --link-dest otoh cares about previous backups.  It assumes you
want to have the old metadata on the old backup(s) and the new
metadata on the new backup.  Since metadata applies to the inode not
the link the only way it can accomplish this is to duplicate the file
instead of linking it.  Therefore, you now have 2 copies of the file
with different ACLs.  This increases disk space usage without
transferring any data.

As for what I do I don't attempt any predictions.  I keep a fixed
number of old backups (different number for different things).  When I
start running low on disk space on my backup system I add more storage.

On 06/14/2014 08:35 PM, François wrote:
> Hi Kevin, hi list,
> 
> First, thanks a lot for your answer.
> 
> I'm not really sure I understand why a cp -al + rsync works where
> rsync alone doesn't, but I gave it a try and it seems it solved my
> issue. Well, mostly :
> 
> What I did :
> 
> 1. cd /path/to/backups 2. cp -al ./backupWithOldACL
> ./backupWithNewACL 2. rsync -ahAXS --numeric-ids source
> /path/to/backups/backupWithNewACL 3. ln -s
> /path/to/backups/backupWithNewACL ./latest 4. Then I ran my script
> again (with --link-dest), which told me that it would need ~150MB
> to run successfully. Since I had ~30GB of free space, it sounded OK
> to me and went to sleep. 5. This morning, I saw in my log that the
> backup completed successfully. But I also saw that the destination
> device is full ! So the backup took a lot more than the 150MB that
> were supposed to be used.
> 
> So, I now think that I am misinterpreting the "Total transferred
> file size" line and that it probably shouldn't be used for what I'm
> trying to do. Can you confirm this ?
> 
> As a result, I'm thinking about changing my strategy to something
> like this : 1. Remove all the dry-run stuff, 2. Run rsync, 3.
> Detect the "device full" error, 4a. If detected, remove the oldest
> backup, 4b. If not, go to 6. 5. Go to 2. 6. Backup successful.
> 
> With your experience and knowledge, could you tell me what you
> think about this ? Is it a better idea than relying on a dry run ?
> 
> Thanks again,
> 
> 
> Le vendredi 13 juin 2014, 17:20:11 Kevin Korb a écrit :
>> The problem here isn't transfer size it is that rsync is storing
>> the affected files with and without the ACL even though the file
>> data is the same.
>> 
>> The solution is to run a single manual rsync using the old method
>> that predates --link-dest...
>> 
>> 1, find your most recent non-ACL backup 2, cp -al thatbackup
>> newbackup 3, rsync -ahAXS --numeric-ids your data over top of the
>> newbackup dir
>> 
>> When you do this rsync will set the ACLs on the existing files
>> (and in every old backup containing the same inode) rather than
>> duplicating the file to store it with and without the ACL.
>> 
>> On 06/13/2014 05:14 PM, daitheflu at free.fr wrote:
>>> Hi,
>>> 
>>> I'm using rsync to backup my files. I have also added a
>>> pre-backup step where the script checks if there is enough
>>> space on the destination for rsync to run successfully. The
>>> goal is to make room (delete oldest backups) if that's not the
>>> case.
>>> 
>>> I'm running rsync with -ahAXS --numeric-ids --link-dest
>>> options.
>>> 
>>> For my pre-backup step, I'm currently using the same options + 
>>> --dry-run --no-h --stats. I then parse the "Total transferred
>>> file size" line, add a 20% (!) padding and compare the result
>>> with the result of a "df" call on destination.
>>> 
>>> It seemd to work properly until I decided to add some ACL on
>>> the source device. I didn't really know how rsync would handle
>>> the situation, but understood that it may have to backup all
>>> the files (instead of hardlinking them) because of this
>>> change.
>>> 
>>> The dry run reported that it would need about 1400 MB and I
>>> have around 30 000 MB of free space, so it should have ran
>>> properly. Sadly, I got the "no space left on device" error.
>>> 
>>> So, is it safe to rely on the stats provided by a dry run to 
>>> achieve my goal ? Does the --dry-run option take care of 
>>> ACL/permissions changes ?
>>> 
>>> 
>>> Thanks a lot for your work and help, rsync is really a
>>> wonderful tool.
>>> 
>>> Cheers,
>> 
>> -- 
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>>
>> 
Kevin Korb			Phone:    (407) 252-6853
>> Systems Administrator		Internet: FutureQuest, Inc.
>> Kevin at FutureQuest.net  (work) Orlando, Florida
>> kmk at sanitarium.net (personal) Web page:
>> http://www.sanitarium.net/ PGP public key available on web site. 
>> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
>
>> 
> 

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlOc66kACgkQVKC1jlbQAQdOzwCff55scNmlKImwGoc6Cj49YWNL
KvkAoILx5jdS8h6qUrXh3ORHo5TDOpbq
=kmuT
-----END PGP SIGNATURE-----