Speed up rsync with many of excluded files

[raf]

Georg Limbach via rsync wrote:

> Hello,
> 
> I have a problem while rsyncing a directory with perhaps 1.000.000
> files. Before this I generate some files locally and add them to temp
> files. So this files should be excluded from syncing.
> 
> My rsync call looks like this:
> 
> rsync --exclude='tmp/' \
> --exclude-from=/tmp/tmp.GF7SsFPnS3 \
> --exclude-from=/tmp/tmp.8SjJNCHyaI \
> --exclude-from=/tmp/tmp.CxZXEoPjgV \
> --exclude-from=/tmp/tmp.G3g2iMo4bs \
> --exclude-from=/tmp/tmp.H9KJYPMfMS \
> --exclude-from=/tmp/tmp.PNi7cJaREP \
> --exclude-from=/tmp/tmp.S4N9H4lsU7 \
> --exclude-from=/tmp/tmp.a5Zlgh6pUK \
> --exclude-from=/tmp/tmp.eiUlMluAe8 \
> --exclude-from=/tmp/tmp.ma0S1YSewc \
> --exclude-from=/tmp/tmp.sLR95oVbVD \
> --exclude-from=/tmp/tmp.zbfeLpezMX \
> -ax --info=progress2 \
> -e 'ssh -x -T -o Compression=no' \
> '/srv/project/shared/uploads/' \
> 'project at host:/srv/project/shared/uploads/'
> 
> In every temp file are 1000s of lines with files like this:
> 
> orders/scan/file/2234480/scan.pdf
> 
> When I started rsync it only should transfer perhaps 1000 files and
> delete 20 ones. But it takes 15 minutes to do that. I think the big
> count of excluded files tear down the compare speed of files.
> 
> What can I do to speed up this process?
> 
> Thanks for your advise!
> 
> Georg

I found a big speed up when I starting providing a list of
candidate files to rsync with the --files-from option.
If you have some way of identifying which files might need
to be rsynced that's quicker than rsync itself checking
everything, it can make a big difference. That's not why
I did it, but it was a nice bonus.

cheers,
raf