--itemize-changes not recursive (not printing created files inside created directories)
Richard Michael
rmichael at edgeofthenet.org
Wed Dec 18 12:11:58 MST 2013
On Wed, Dec 18, 2013 at 1:21 PM, Kevin Korb <kmk at sanitarium.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> - -c checksums everything *BEFORE* transferring anything. It would only
> validate an rsync if run *AFTER* an rsync has already finished and
> nothing has changed since the rsync ran.
Yes, I'm fairly sure I understand. I'm not using it to validate the
transfer; I'm using it to decide whether or not to transfer. I'll
compare the transfer list with a "regular" heuristic rsync afterward.
>
> Also, rsync's -c is rather dumb as it computes checksums for files
> that have different sizes so they can't possibly be the same and it
> computes checksums for files that only exist on one end and therefore
> has nothing to compare them to.
Indeed, those seem like cases which could be optimized someday.
>
> On 12/18/2013 11:45 AM, Richard Michael wrote:
>> Kevin,
>>
>> Thanks for your reply.
>>
>> On Tue, Dec 17, 2013 at 6:49 PM, Kevin Korb <kmk at sanitarium.net>
>> wrote: You are running with -n (--dry-run) It isn't logging any
>> transfers because it isn't actually doing any. Your command should
>> show the dry run details on stdout but only what it actually
>> transferred (nothing) in the log file. If you drop the -n it will
>> actually make the changes and log them.
>>
>>> Ah yes, I should have checked this, thanks!
>>
>>> I should prepare a tiny man page patch mentioning --dry-run will
>>> affect --log-file output.
>>
>>> I suppose for my use-case, I should ignore --log-file and simply
>>> capture stdout, because they are nearly identical except for the
>>> misleading missing transfers:
>>
>>> $ diff -B <(cut -d' ' -f 4- rsync.test1.log) rsync.test1.stdout
>>> 1c1 < building file list ---
>> sending incremental file list
>>> 8a9
>>>>> f+++++++++
>>>>> S-1-5-21-947408474-2404634453-201028329-3641/desktop.ini
>>> 9a11
>>>>> f+++++++++
>>>>> S-1-5-21-947408474-2404634453-201028329-3642/desktop.ini
>>> 10a13
>>>>> f+++++++++
>>>>> S-1-5-21-947408474-2404634453-201028329-500/desktop.ini
>>
>>
>> Note also that you shouldn't use -c unless you have a really good
>> (and unusual) reason.
>>
>>> "shouldn't" ? I'm aware it will take longer, and don't mind.
>>> I'm thoroughly testing an rsync of two NTFS filesystems, and want
>>> to avoid any time precision issues on at least one test run.
>>
>>
>>> Regards, Richard
>>
>>
>>
>> On 12/17/2013 06:19 PM, Richard Michael wrote:
>>>>> Hello list,
>>>>>
>>>>> Using rsync version 3.1.0pre1 protocol version 31.PR14, I
>>>>> ran this command to sync two local filesystems:
>>>>>
>>>>> rsync -rvvctpgonh --itemize-changes --progress --stats
>>>>> --log-file=/root/rsync.log /mnt/ntfs/vmdk/
>>>>> /mnt/ntfs/restore/
>>>>>
>>>>> -r = recursive -vv = verbose -c = checksum compare -t =
>>>>> transfer modification times -p = set destination perms to
>>>>> match source -g = set destination group to match source -o =
>>>>> set destination owner to match source -n = dry run (should
>>>>> cause --progress to be meaningless, but harmless) -h = human
>>>>> readable output
>>>>>
>>>>>
>>>>> Output includes this summary:
>>>>>
>>>>> 2013/12/11 16:35:14 [3349] Number of files: 443,967 (reg:
>>>>> 419,427, dir: 24,540) 2013/12/11 16:35:14 [3349] Number of
>>>>> created files: 2,581 (reg: 2,511, dir: 70) 2013/12/11
>>>>> 16:35:14 [3349] Number of deleted files: 0 2013/12/11
>>>>> 16:35:14 [3349] Number of regular files transferred: 2,744
>>>>> 2013/12/11 16:35:14 [3349] Total file size: 560.66G bytes
>>>>> 2013/12/11 16:35:14 [3349] Total transferred file size:
>>>>> 196.26G bytes 2013/12/11 16:35:14 [3349] Literal data: 0
>>>>> bytes 2013/12/11 16:35:14 [3349] Matched data: 0 bytes
>>>>> 2013/12/11 16:35:14 [3349] File list size: 589.75K 2013/12/11
>>>>> 16:35:14 [3349] File list generation time: 0.047 seconds
>>>>> 2013/12/11 16:35:14 [3349] File list transfer time: 0.000
>>>>> seconds 2013/12/11 16:35:14 [3349] Total bytes sent: 20.97M
>>>>> 2013/12/11 16:35:14 [3349] Total bytes received: 1.36M
>>>>> 2013/12/11 16:35:14 [3349] sent 20.97M bytes received 1.36M
>>>>> bytes 2.28K bytes/sec 2013/12/11 16:35:14 [3349] total size
>>>>> is 560.66G speedup is 25,111.56 (DRY RUN)
>>>>>
>>>>>
>>>>> --itemize-changes has output 70 lines pertaining to created
>>>>> directories (matches the summary output above), similar to
>>>>> this:
>>>>>
>>>>> 2013/12/11 13:54:23 [3349] cd+++++++++
>>>>> $RECYCLE.BIN/S-1-5-21-947408474-2404634453-201028329-3641/
>>>>>
>>>>> $ awk '$4 ~ /^cd/' rsync.log | wc -l # ==> 70
>>>>>
>>>>> However, the log file contains 0 occurrences of '>', '<' or
>>>>> 'cf++++++++' .
>>>>>
>>>>>
>>>>> A bit of arithmetic indicates exactly the "Number of regular
>>>>> files transferred" ( = 2,744 from stats output) are missing
>>>>> from the log:
>>>>>
>>>>> $ wc -l rsync.log # ==> 441279 ( - 56 lines of stats/info +
>>>>> 2,744 unreported files = 443967 files/dirs, as summarized)
>>>>>
>>>>>
>>>>> Questions:
>>>>>
>>>>> 1/ Why doesn't --itemize-changes report "cf+++++++++" lines
>>>>> for files contained inside created directories ("cd+++++++")?
>>>>> For example, I expected output similar to: (faked output)
>>>>>
>>>>> 2013/12/11 13:54:23 [3349] cd+++++++++
>>>>> $RECYCLE.BIN/S-1-5-21-947408474-2404634453-201028329-3641/
>>>>> 2013/12/11 13:54:23 [3349] cf+++++++++
>>>>> $RECYCLE.BIN/S-1-5-21-947408474-2404634453-201028329-3641/desktop.ini
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
> 2/ Of the 2,744 transferred files, 2,581 are created (I expect
>>>>> each, dir or file, as a "c[fd]+++++++") ; this leaves 163
>>>>> transferred files, presumably because they were changed (but
>>>>> not created), also not reported. Why were they not
>>>>> reported?
>>>>>
>>>>>
>>>>> Ultimately, I would like a list of all differences, both file
>>>>> and directory, sent to the client. I suspect
>>>>> --itemize-changes is a bit archaic, and I should be
>>>>> investigating --info= customized output.
>>>>>
>>>>> Thank you!
>>>>>
>>>>> Regards, Richard
>>>>>
>>
>>> -- Please use reply-all for most replies to avoid omitting the
>>> mailing list. To unsubscribe or change options:
>>> https://lists.samba.org/mailman/listinfo/rsync Before posting,
>>> read: http://www.catb.org/~esr/faqs/smart-questions.html
>
> - --
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
> Kevin Korb Phone: (407) 252-6853
> Systems Administrator Internet:
> FutureQuest, Inc. Kevin at FutureQuest.net (work)
> Orlando, Florida kmk at sanitarium.net (personal)
> Web page: http://www.sanitarium.net/
> PGP public key available on web site.
> ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.22 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEARECAAYFAlKx57kACgkQVKC1jlbQAQfgYACfZDIKobBo0vO8BsFdn7YOesvM
> RcgAoMORYlmTmoZiof4IIioyGDhtw7H+
> =eKNH
> -----END PGP SIGNATURE-----
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
More information about the rsync
mailing list