Very surprising behaviour with --files-from

Robin Lee Powell rlpowell at
Fri Dec 10 10:11:39 MST 2010

$ wc -l /tmp/list
1000 /tmp/list

$ rsync -i -aPv --ignore-existing --files-from=/tmp/list /backups/ ut00-s00010:/backups/
building file list ...
3937 files to consider

I am totally baffled.

That's not such a big deal, but the list I'm *actually* using has
twenty *million* files in it.  At a couple hundred files a second,
if it's going to check 4 times the number of files, that's a *huge*
time waste.  What's going on?

Here's what the list looks like:

$ head /tmp/list

They are all like that, and they are all files, not directories.

I thought it could be rsync checking the directories that have those
files in them, but there are only 4300 directories, and when I
stopped the big version (OK, *that* was a mistake, but I was worried
about the behaviour) it was saying "28395900 files...", which is
rather a lot more than 20 million + 4300.

This is making a many hours difference to an already very long
process; anyone know what's going on?


-- :  Our last, best hope for a fantastic future.
Lojban ( The language in which "this parrot
is dead" is "ti poi spitaki cu morsi", but "this sentence is false"
is "na nei".   My personal page:

More information about the rsync mailing list