Verifying backups

Kevin Korb kmk at sanitarium.net
Wed Sep 30 21:29:29 UTC 2015


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

reply inline...

On 09/30/2015 05:18 PM, Ronald F. Guilmette wrote:
> 
> In message <560C4A51.4040702 at sanitarium.net>, Kevin Korb
> <kmk at sanitarium.net> wrote:
> 
>> First off, --fileflags --force-change are not in my man rsync so
>> I don't know what those are.
> 
> These are probably (Free)BSD specific.  Here's what the man page
> says:
> 
> --fileflags             preserve file-flags (aka chflags) 
> --force-change          affect user/system immutable files/dirs

Interesting, these aren't on my FreeBSD system.

>> Second, you should look into using either ZFS subvolume snapshots
>> or rsync --link-dest to maintain multiple backups.
> 
> Thank you, but I have no real interest in switching to ZFS just
> now.
> 
>> Now, for your actual question... Add --itemize-changes to your
>> standard command line.  -v is almost entirely useless without it
>> anyway.
> 
> Thanks!  I certainly did not know about that option!
> 
>> To figure out and fix what is corrupt you have 2 paths:
>> 
>> 1. Add --checksum for a single pass.  This will take forever as
>> rsync checksums everything even things it shouldn't expect to
>> match (even things that are only on one end!).  Anything that
>> checksum finds that rsync wouldn't have otherwise found would
>> have a 'c' but not a 't'.
> 
> I don't understand what you mean about 'c' and 't'.  Are you
> talking about what will appear in the (itemized) change log?  I am
> guessing so.

You will when you run sync with --itemize-changes.  It prepends a
string to the file name list telling you what is different about a
file. A c without a t means that the file had a different checksum but
not a different timestamp which shouldn't happen.

> 
>> 2.  Add --ignore-times for a single pass.
> 
> NOW I am REALLY confused!  I don't understand at all what the
> functional difference is between these two options:
> 
> -c, --checksum              skip based on checksum, not mod-time &
> size -I, --ignore-times          don't skip files that match size
> and time

- --checksum == copy only things that have a different checksum and
ignore everything else

- --ignore-times == assume everything is wrong and re-delta-copy it all.
 If you aren't using --whole-file then only the different parts of the
file is copied.  If you are using --whole-file then it just means
re-copy everything.

> 
> Could you please explain?  Both of these options would seem to me
> to have exactly the same/identical effect.
> 
>> ...  Normally this doesn't take as long as --checksum.  However,
>> since you are using an external USB device which means you are
>> also using --whole-file...
> 
> No, I am *not* using the --whole-file option.  Indeed, up until
> this moment, I didn't even know that option existed!  I thank you
> for bringing it to my attention.  Now I plan to be using
> --whole-file in future when making all of my backups.  (Because
> most of the files I back up are binary media files, I think that
> the delta algorithm is not really saving me that much in terms of
> run-time.)
> 
> Having said that however, I need also to say that your comment
> (just above) is, for me, puzzling and downright cryptic.  Yes, I am
> doing my backups to an external USB-connected device.  But what has
> that fact got to do with the --whole-file option?  I see no
> obvious connection at all.

If your source and target are both local then --whole-file is forced.
 This is a performance feature as delta-transferring a file locally is
a ton of extra work for no benefit at all.

With --whole-file the local copy process is:
read file from source
write file to target

Without --whole-file the process would be:
read file from source and hash the pieces of it
read file from target and hash the pieces of it
compare the hashes
if difference
  if --inplace
    write file to target beginning at difference
  else
    write entire file to target



> 
>> ... this will probably be even slower than --checksum.
> 
> Why would using the --whole-file option during my attempts to
> verify my backup files cause things to run even slower?  (This is
> not at all obvious.)
> 
>> Either way, you need to get your system writing files correctly.
> 
> Oh yes, clearly.  I believe (for now) that simply having proper
> cooling applied to the external drive in question may be sufficient
> to prevent corruption of files put on the drive in futire.  (I did
> run one of the LONG built-in drive self-test passes on this drive
> after I found that some files on it had been corrupted, but results
> from that were 100% OK. So I think the drive perhaps just got a bit
> flaky during a time when I *did not* have an small utility fan
> right next to it and pointing at it. I _do_ have that now.)
> 
> Thank you for all of your detailed responses.
> 
> 
> Regards, rfg
> 

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
- -*~
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
- -*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlYMVDkACgkQVKC1jlbQAQfNygCfaivAGfISYTJ4a/8HWNZVjBD8
704AoJhfyvS8ycX8piw4453lUZRtJP0N
=EQVA
-----END PGP SIGNATURE-----



More information about the rsync mailing list