Skip based on checksum not worked as expected when using with complex filter rules.

[Kevin Korb]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Any time rsync wants to do something to a file you don't think it
should be see what --itemize-changes says about it.  In fact, -v is
almost entirely useless without --itemize-changes.  In fact, if I was
in charge I would make -v include --itemize-changes.

On 04/17/2015 09:21 PM, Hongyi Zhao wrote:
> Hi all,
> 
> I've a local repository for Debian, and I'm doing some rsync
> testings on it.  See the following commands for detail:
> 
> werner at debian:~/software/LocalRepo$ rsync -amvHKLPRSB131072 -n \ 
> ftp.uk.debian.org::debian/./dists/wheezy-updates/main/\ 
> debian-installer/binary-amd64/Packages.gz \ 
> repo.debian.local/debian/ [snip] 
> dists/wheezy-updates/main/debian-installer/binary-amd64/Packages.gz
>
>  sent 46 bytes  received 202 bytes  55.11 bytes/sec total size is
> 20  speedup is 0.08 (DRY RUN)
> 
> werner at debian:~/software/LocalRepo$ rsync -amvHKLPRSB131072 -nc \ 
> ftp.uk.debian.org::debian/./dists/wheezy-updates/main/\ 
> debian-installer/binary-amd64/Packages.gz \ 
> repo.debian.local/debian/ [snip] sent 46 bytes  received 218 bytes
> 58.67 bytes/sec total size is 20  speedup is 0.08 (DRY RUN)
> 
> As you can see, if I use checksum on that package, it shown that
> it's just the same to my local copy, so there is not need to be
> rsynced.
> 
> But when I use a complex rsync filter rules script based on
> checksum and others to check all of the stuff based on the
> corresponding rules, it shows that the above package need to be
> updated, see the following for detail:
> 
> 1- My script is named as su_dists.rsync and the content of it is as
>  follows, sorry for its complex ;-(
> 
> ___________ begin su_dists.rsync ____________
> 
> rsync -amvHKLPRSB131072 -nc --delete --delete-excluded \ -f
> +_dists/wheezy/**Translation-en* \ -f
> +_dists/wheezy/**binary-all/Packages.gz \ -f
> +_dists/wheezy/Release* \ -f
> P_dists/wheezy-backports/**Translation-en* \ -f
> P_dists/wheezy-backports/**binary-all/Packages.gz \ -f
> P_dists/wheezy-backports/Release* \ -f
> P_dists/wheezy-proposed-updates/**Translation-en* \ -f
> P_dists/wheezy-proposed-updates/**binary-all/Packages.gz \ -f
> P_dists/wheezy-proposed-updates/Release* \ -f
> +_dists/wheezy-updates/**Translation-en* \ -f
> +_dists/wheezy-updates/**binary-all/Packages.gz \ -f
> +_dists/wheezy-updates/Release* \ -f
> +_dists/wheezy/**binary-amd64/Packages.gz \ -f
> +_dists/wheezy/**installer-amd64/current/*** \ -f
> P_dists/wheezy-backports/**binary-amd64/Packages.gz \ -f
> P_dists/wheezy-backports/**installer-amd64/current/*** \ -f
> P_dists/wheezy-proposed-updates/**binary-amd64/Packages.gz \ -f
> P_dists/wheezy-proposed-updates/**installer-amd64/current/*** \ -f
> +_dists/wheezy-updates/**binary-amd64/Packages.gz \ -f
> +_dists/wheezy-updates/**installer-amd64/current/*** \ -f
> +_dists/wheezy/**binary-i386/Packages.gz \ -f
> +_dists/wheezy/**installer-i386/current/*** \ -f
> P_dists/wheezy-backports/**binary-i386/Packages.gz \ -f
> P_dists/wheezy-backports/**installer-i386/current/*** \ -f
> P_dists/wheezy-proposed-updates/**binary-i386/Packages.gz \ -f
> P_dists/wheezy-proposed-updates/**installer-i386/current/*** \ -f
> +_dists/wheezy-updates/**binary-i386/Packages.gz \ -f
> +_dists/wheezy-updates/**installer-i386/current/*** \ -f
> P_dists/jessie/*** \ -f P_dists/jessie-backports/*** \ -f
> P_dists/jessie-proposed-updates/*** \ -f P_dists/jessie-updates/***
> \ -f P_pool/*** \ -f +_dists/***/ \ -f -_*
> ftp.uk.debian.org::debian/./ ./repo.debian.local/debian/
> 
> ___________ end su_dists.rsync ____________
> 
> 2- Then, I run my above script as follows to obtain all of the
> files filtered-out according to my filter rules and checkout the
> ones need to be updated based on checksum:
> 
> werner at debian:~/software/LocalRepo$ bash su_dists.rsync|awk -F'/'
> \ '/^dists/ && $NF' dists/wheezy-updates/Release 
> dists/wheezy-updates/Release.gpg 
> dists/wheezy-updates/contrib/binary-i386/Packages.gz 
> dists/wheezy-updates/contrib/debian-installer/binary-amd64/Packages.gz
>
> 
dists/wheezy-updates/contrib/debian-installer/binary-i386/Packages.gz
> dists/wheezy-updates/main/debian-installer/binary-amd64/Packages.gz
>
> 
dists/wheezy-updates/main/debian-installer/binary-i386/Packages.gz
> dists/wheezy-updates/non-free/binary-amd64/Packages.gz 
> dists/wheezy-updates/non-free/binary-i386/Packages.gz 
> dists/wheezy-updates/non-free/debian-installer/binary-amd64/Packages.gz
>
> 
dists/wheezy-updates/non-free/debian-installer/binary-i386/Packages.gz
> dists/wheezy-updates/non-free/i18n/Translation-en.bz2 
> dists/wheezy/contrib/debian-installer/binary-all/Packages.gz 
> dists/wheezy/contrib/debian-installer/binary-amd64/Packages.gz 
> dists/wheezy/contrib/debian-installer/binary-i386/Packages.gz 
> dists/wheezy/non-free/debian-installer/binary-all/Packages.gz 
> dists/wheezy/non-free/debian-installer/binary-amd64/Packages.gz 
> dists/wheezy/non-free/debian-installer/binary-i386/Packages.gz
> 
> As you can see, this time, the following file is considered to be
> updated:
> 
> dists/wheezy-updates/main/debian-installer/binary-amd64/Packages.gz
>
>  But it has been considered not need to be updated in my first run
> with the -c/--checksum option from the terminal other than the bash
> srcipt.
> 
> 3- If I change my bash script into the following form, i.e., with
> the `-- no-t' option added to the args list:
> 
> ___________ begin su_dists.rsync ____________
> 
> rsync -amvHKLPRSB131072 -nc --no-t --delete --delete-excluded \ -f
> +_dists/wheezy/**Translation-en* \ -f
> +_dists/wheezy/**binary-all/Packages.gz \ -f
> +_dists/wheezy/Release* \ -f
> P_dists/wheezy-backports/**Translation-en* \ -f
> P_dists/wheezy-backports/**binary-all/Packages.gz \ -f
> P_dists/wheezy-backports/Release* \ -f
> P_dists/wheezy-proposed-updates/**Translation-en* \ -f
> P_dists/wheezy-proposed-updates/**binary-all/Packages.gz \ -f
> P_dists/wheezy-proposed-updates/Release* \ -f
> +_dists/wheezy-updates/**Translation-en* \ -f
> +_dists/wheezy-updates/**binary-all/Packages.gz \ -f
> +_dists/wheezy-updates/Release* \ -f
> +_dists/wheezy/**binary-amd64/Packages.gz \ -f
> +_dists/wheezy/**installer-amd64/current/*** \ -f
> P_dists/wheezy-backports/**binary-amd64/Packages.gz \ -f
> P_dists/wheezy-backports/**installer-amd64/current/*** \ -f
> P_dists/wheezy-proposed-updates/**binary-amd64/Packages.gz \ -f
> P_dists/wheezy-proposed-updates/**installer-amd64/current/*** \ -f
> +_dists/wheezy-updates/**binary-amd64/Packages.gz \ -f
> +_dists/wheezy-updates/**installer-amd64/current/*** \ -f
> +_dists/wheezy/**binary-i386/Packages.gz \ -f
> +_dists/wheezy/**installer-i386/current/*** \ -f
> P_dists/wheezy-backports/**binary-i386/Packages.gz \ -f
> P_dists/wheezy-backports/**installer-i386/current/*** \ -f
> P_dists/wheezy-proposed-updates/**binary-i386/Packages.gz \ -f
> P_dists/wheezy-proposed-updates/**installer-i386/current/*** \ -f
> +_dists/wheezy-updates/**binary-i386/Packages.gz \ -f
> +_dists/wheezy-updates/**installer-i386/current/*** \ -f
> P_dists/jessie/*** \ -f P_dists/jessie-backports/*** \ -f
> P_dists/jessie-proposed-updates/*** \ -f P_dists/jessie-updates/***
> \ -f P_pool/*** \ -f +_dists/***/ \ -f -_*
> ftp.uk.debian.org::debian/./ ./repo.debian.local/debian/
> 
> ___________ end su_dists.rsync ____________
> 
> This time, I will obtain the correct file-lists that should be
> updated based on the checksum checking, see following for detail:
> 
> werner at debian:~/software/LocalRepo$ bash su_dists.rsync|awk -F'/'
> \ '/^dists/ && $NF' dists/wheezy-updates/Release 
> dists/wheezy-updates/Release.gpg 
> dists/wheezy-updates/non-free/i18n/Translation-en.bz2
> 
> As you can see, for my case, there only have three files which need
> to be updated all toghether at the moment.
> 
> Any hints on the above issues?
> 
> Regards
> 

- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iEYEARECAAYFAlUxsxwACgkQVKC1jlbQAQcTNQCgnjBUW0D3nS0e7y/HL5lQozyw
eGIAnjokKS3M+U0IM2xKUy9xSb/n5rUa
=ZLxY
-----END PGP SIGNATURE-----