Rsync'ing lists of files

[Alberto Accomazzi]

Just so that we don't forget the lessons from the past, let me point
out that we had discussion and testing done on this subject back in
November, with mixed results (i.e. YMMV):
   http://lists.samba.org/pipermail/rsync/2001-November/005398.html
I think the consensus from that experiment was that implementing the
option using the include/exclude mechanism was not the way to go
(correct me if I'm wrong Dave).

Andrew Schorr's patch does this differently but from what I can tell
it would only work when uploading files to a server (which is the 
opposite of what my experiments with --files-from were):
   http://lists.samba.org/pipermail/rsync/2001-November/005272.html

Since it seems that different people want this option for different
purposes, we need to make sure that it gets implemented in a sensible
way, with some testing being done to ensure that we still have decent
performance and that it works in all cases (sending/receiving files).

So I think our strategy should be to bug Dave Dykstra until he gives
up and writes the patch :-)

-- Alberto


Stephane Paltani wrote:

> Dave Dykstra wrote:
> > 
> > Sigh, another request for the --files-from I promised to write over 6
> > months ago, but I've been so overloaded at work lately that I don't know if
> > I'm ever going to get to it.  Perhaps someone else will have to do it.
> 
> He he, happy to see a general consensus for this feature!
> 
> > It turns out that back in rsync 2.3.2 and earlier there was an optimization
> > (which I wrote and actually was the primary reason that I volunteered to be
> > maintainer of rsync for a while) that kicked in when there was list of
> > includes with no wildcards followed by an --exclude '*', and there was no
> > --delete.  Instead of recursing through the files and doing comparisons, it
> > would just directly open the files in the include list.  It only had to be
> on the sending side, you might want to try 2.3.2 on your sending side to
> > see if you get a significant performance boost.  Andrew Tridgell took it
> > out in 2.4.0 because he didn't like how it changed the usual semantics of
> > requiring all parent directories to be explicitly listed in the include
> > list.
> 
> Whoops! That did the trick for me! It took me 6 minutes to transfer 250 GB!
> Too bad it has been turned down. I have the impression it would satisfy
> most, if not all, "--files-from" lobbyists.


****************************************************************************
Alberto Accomazzi                          mailto:aaccomazzi at cfa.harvard.edu
NASA Astrophysics Data System                      http://adsabs.harvard.edu
Harvard-Smithsonian Center for Astrophysics        http://cfawww.harvard.edu
60 Garden Street, MS 83, Cambridge, MA 02138 USA   
****************************************************************************