rsync very slow with large include/exclude file list

ray vantassle rayvantassle at gmail.com
Mon Jun 15 16:27:53 MDT 2015


I have a sensor collector system (very low-powered slow ARM cpu), and
another system which daily pulls the data files from it for processing.
There are about 1000 new files each day.  As part of the processing it
decides that certain of the files are of no interest, and adds them to an
exclude file, which is used in future rsyncs.  No files are ever deleted
from the source system, just the receiver system.  All done in cron jobs at
night.

After about 18 months, there are about 350,000 files and the exclude list
has about 72,000 filenames.  I recently ran the pulling script manually and
thought the system must have died.
Rsync took almost 3 hours.  Trying to narrow down the problem, I removed
the "--exclude-file=" option from the rsync command -- and it took only 16
minutes -- including the time to transfer the 72,000 files that are of no
interest.
 (con't)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20150615/9257f573/attachment.html>


More information about the rsync mailing list