Very surprising behaviour with --files-from
Robin Lee Powell
rlpowell at digitalkingdom.org
Fri Dec 10 10:11:39 MST 2010
$ wc -l /tmp/list
$ rsync -i -aPv --ignore-existing --files-from=/tmp/list /backups/ ut00-s00010:/backups/
building file list ...
3937 files to consider
I am totally baffled.
That's not such a big deal, but the list I'm *actually* using has
twenty *million* files in it. At a couple hundred files a second,
if it's going to check 4 times the number of files, that's a *huge*
time waste. What's going on?
Here's what the list looks like:
$ head /tmp/list
They are all like that, and they are all files, not directories.
I thought it could be rsync checking the directories that have those
files in them, but there are only 4300 directories, and when I
stopped the big version (OK, *that* was a mistake, but I was worried
about the behaviour) it was saying "28395900 files...", which is
rather a lot more than 20 million + 4300.
This is making a many hours difference to an already very long
process; anyone know what's going on?
http://singinst.org/ : Our last, best hope for a fantastic future.
Lojban (http://www.lojban.org/): The language in which "this parrot
is dead" is "ti poi spitaki cu morsi", but "this sentence is false"
is "na nei". My personal page: http://www.digitalkingdom.org/rlp/
More information about the rsync