Incremental file-list recursion has landed in CVS

[Wayne Davison]

On Thu, Jan 11, 2007 at 05:22:55PM -0500, Matt McCutchen wrote:
> Specifically, I'm curious about what areas under the source
> argument(s) are scanned at what time.

All the args that the user supplies are scanned at once, allowing them
to be unduplicated as they would be in a normal transfer.  The only
difference is that no recursing happens during the initial sending of
the first file-list.  Then, one directory at a time is scanned and sent
over until we have a decent number of upcoming files in the pipeline for
the generator.  The result is that typical transfers with a small number
of source args can start transferring files almost immediately, and the
depth-first scan of directories continues intermixed in with the file
transfers (or at least intermixed with the generator's scanning for
changed files when no transfers are needed).

If the --relative option is used, implied directories are treated as
"args" and sent in the first wave.  The --files-from option has always
treated the items read from the file as args, so a transfer with a huge
number of files-from items and no real recursion doesn't get any benefit
from the incremental recursion.

> Also, does the incremental scan rule out "file has vanished" warnings?

It lessens their chance of occurring because the time that elapses
between a directory scan and the time the generator starts to work on
those files is much shorter than waiting for the full scan to complete.
However, there can still be vanished files as there is still some
reading ahead of directories (rsync tries to keep a good amount of work
in the pipeline for the generator to blaze through).  I haven't decided
exactly what I want the read-ahead limit to be, but the current code
wants 1000 files to be available beyond the currently-active directory.

..wayne..