rsync 2.5.6 globbing bug

[jw schultz]

On Wed, Jul 30, 2003 at 11:39:06PM +0100, Alan Burlison wrote:
> Wayne Davison wrote:
> 
> >The best way to do this is to use a starting dir, the --recursive
> >option, and then exclude all subdirs:
> >
> >rsync -av exclude='*/' host::module/path/ .
> 
> That works (thanks), but isn't exactly intuitive ;-)
> 
> >Since the arg-list created for file globbing is allocated memory extra
> >to the file list that will be created for the transfer, it's certainly
> >more efficient to use the aforementioned exclusion method.  The current
> >limit of 1000 files seems rather arbitrary, but I'm not sure that it
> >really needs to be made larger than this (given that there are ways to
> >work around the limit).  You can feel free to disagree, if you like --
> >let me know your reasons, if you do.
> 
> Hew so more efficient?  The glob code doesn't have to descend into any 
> subdirs, and will only expand to the list of required files, rather than 
> having to expand the list then filter it with excludes.

You obviously don't know how globbing or the exclude list
works.  Excluding like this is more efficient than globbing.
In terms of CPU time, memory and the logic that requires the
subdirs be skipped because they are dirs and we you aren't
recursing.  On the other hand, efficiency at this
particular point it the code is largely irrelevant.  What is
relevant is sanity.

> If you are really worried about memory you could malloc say a 64-entry argv 
> and realloc by powers of 2 - that would actually use *less* memory than the 
> current mechanism in the default case of relatively small directories.
> 
> The root of my concern is that if people have a large leaf directory they 
> wouldn't expect to have to specify --recurse and --exclude just to copy the 
> contents, and the diagnostics returned when it fails are completely opaque 
> - in this case I had to resort to a debugger to find out what the root 
> cause was.  I can't see that it is reasonable to expect to have to debug 
> rsync to find out that my directory is too big for rsync to cope with.
> 
> The problem can be made to go away entirely with a relatively code minor 
> change, which will be one less cause of the "rsync: connection unexpectedly 
> closed" problems that people hit so often, and in addition I'm happy to 
> submit a patch, so minimal work is involved for anyone else.

Eliminating the limit is not a minor code change.  This is
such a rare case it isn't funny.  It is rare to have that
many files in a directory.  It is even more rare for someone
to want all the files in such a directory but not any
subdirectories.

-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw at pegasys.ws

		Remember Cernan and Schmitt