Speed up rsync with many of excluded files

Georg Limbach georf at georf.de
Tue Nov 12 19:40:07 UTC 2019


Hello,

I have a problem while rsyncing a directory with perhaps 1.000.000
files. Before this I generate some files locally and add them to temp
files. So this files should be excluded from syncing.

My rsync call looks like this:

rsync --exclude='tmp/' \
--exclude-from=/tmp/tmp.GF7SsFPnS3 \
--exclude-from=/tmp/tmp.8SjJNCHyaI \
--exclude-from=/tmp/tmp.CxZXEoPjgV \
--exclude-from=/tmp/tmp.G3g2iMo4bs \
--exclude-from=/tmp/tmp.H9KJYPMfMS \
--exclude-from=/tmp/tmp.PNi7cJaREP \
--exclude-from=/tmp/tmp.S4N9H4lsU7 \
--exclude-from=/tmp/tmp.a5Zlgh6pUK \
--exclude-from=/tmp/tmp.eiUlMluAe8 \
--exclude-from=/tmp/tmp.ma0S1YSewc \
--exclude-from=/tmp/tmp.sLR95oVbVD \
--exclude-from=/tmp/tmp.zbfeLpezMX \
-ax --info=progress2 \
-e 'ssh -x -T -o Compression=no' \
'/srv/project/shared/uploads/' \
'project at host:/srv/project/shared/uploads/'

In every temp file are 1000s of lines with files like this:

orders/scan/file/2234480/scan.pdf

When I started rsync it only should transfer perhaps 1000 files and
delete 20 ones. But it takes 15 minutes to do that. I think the big
count of excluded files tear down the compare speed of files.

What can I do to speed up this process?

Thanks for your advise!

Georg



More information about the rsync mailing list