Help with complicated heirarchy exclude syntax

[Matt McCutchen]

On Sat, 2007-12-22 at 17:02 -0600, reader at newsguy.com wrote:
> One further aspect of this that would be nice to solve too, is that
> using that technique in this case means that rsync has to check
> something like 550,000 (literally) messages to get thru a sync.
> 
> That number will dwindle to a fraction of that eventually but it will
> be a while, and of course I need dayly backups of the other stuff. 
> 
> My test run has been running since I read mail reply.. probably 15
> minutes so far.
> 
> I'm guessing there isn't much of any way around that... and just have
> schedule the backups accordingly.

Right, there's certainly no way to avoid considering all the source
files, and no easy way to avoid considering excluded files (to discover
that they are excluded).  But do try the incremental recursion mode in
development versions of rsync 3.0.0 if you haven't already; it will
reduce memory usage and help streamline the process.

Incidentally, if the sender could fill the socket with file data and
then go incrementally wading through the excluded files while it is
waiting on the network, that might *really* help.  This enhancement
would be easier to make if the complex I/O control flow in each process
were refactored into coroutines that trade control at defined points
when they block.  I wonder if there is a good coroutine library for C
that could handle such things as selecting on the combined set of fds
when all coroutines block and resuming the coroutine whose fd became
ready.

Matt