Speed up rsync with many of excluded files

[raf]

raf via rsync wrote:

> Georg Limbach via rsync wrote:
> 
> > Hello,
> > 
> > I have a problem while rsyncing a directory with perhaps 1.000.000
> > files. Before this I generate some files locally and add them to temp
> > files. So this files should be excluded from syncing.
> > 
> > My rsync call looks like this:
> > 
> > rsync --exclude='tmp/' \
> > --exclude-from=/tmp/tmp.GF7SsFPnS3 \
> > --exclude-from=/tmp/tmp.8SjJNCHyaI \
> > --exclude-from=/tmp/tmp.CxZXEoPjgV \
> > --exclude-from=/tmp/tmp.G3g2iMo4bs \
> > --exclude-from=/tmp/tmp.H9KJYPMfMS \
> > --exclude-from=/tmp/tmp.PNi7cJaREP \
> > --exclude-from=/tmp/tmp.S4N9H4lsU7 \
> > --exclude-from=/tmp/tmp.a5Zlgh6pUK \
> > --exclude-from=/tmp/tmp.eiUlMluAe8 \
> > --exclude-from=/tmp/tmp.ma0S1YSewc \
> > --exclude-from=/tmp/tmp.sLR95oVbVD \
> > --exclude-from=/tmp/tmp.zbfeLpezMX \
> > -ax --info=progress2 \
> > -e 'ssh -x -T -o Compression=no' \
> > '/srv/project/shared/uploads/' \
> > 'project at host:/srv/project/shared/uploads/'
> > 
> > In every temp file are 1000s of lines with files like this:
> > 
> > orders/scan/file/2234480/scan.pdf
> > 
> > When I started rsync it only should transfer perhaps 1000 files and
> > delete 20 ones. But it takes 15 minutes to do that. I think the big
> > count of excluded files tear down the compare speed of files.
> > 
> > What can I do to speed up this process?
> > 
> > Thanks for your advise!
> > 
> > Georg
> 
> I found a big speed up when I starting providing a list of
> candidate files to rsync with the --files-from option.
> If you have some way of identifying which files might need
> to be rsynced that's quicker than rsync itself checking
> everything, it can make a big difference. That's not why
> I did it, but it was a nice bonus.

But it might not help if doing --delete at the same
time. Can the --delete be done occasionally instead of
every time? Unless the files to be deleted can be added
to the list of files handed to rsync. I don't know if
that'll do what I think but it might.

cheers,
raf