specifying a list of files to transfer

[jw schultz]

On Thu, Jan 16, 2003 at 05:07:13PM -0800, Wayne Davison wrote:
> > On Wed, Jan 15, 2003 at 02:49:08PM -0800, jw schultz wrote:
> > > You seem to see the files-from as a way of replacing command-line
> > > args where i see it as a way of replacing the tree scan.
> 
> I actually think of it as both, since I also consider the command-line
> as a way of replacing the tree scan.  I think that it is fairly easy to
> explain how --files-from works if you explain it in terms of how it is
> much like specifying the names on the command-line (and explain what is
> different about it).  One thing that I'm hoping to avoid is arbitrary
> limits on what the mode can do -- I'd like to see it be an easy way to
> specify exactly what files to send, and also as a way to extend the size
> of the command-line.

The difference seems pretty big to me but if you can
describe it cleanly that's fine.  The big thing is that
paths specified in --files-from are relative to the tree,
not CWD and that entries therein are not flattened like they
would be on the command-line (sans -R).


> On Thu, Jan 16, 2003 at 02:52:38PM -0600, Dave Dykstra wrote:
> > I would rather have the '-r' option ignored when --files-from is in
> > effect.
> 
> I wouldn't want that as a hard limit.  It would be better to say that
> -a doesn't imply -r when --files-from is used, but the user can still
> manually specify -r if they want to.  It would be easy to implement this
> in a way that would _not_ require any particular order to the options on
> the command-line (which I agree with JW would be a very bad idea).
> 
> > If people leave out the directories, missing parent directories should
> > be automatically created.
> 
> (I assume you're talking about when using -R, which is not currently on
> by default.)  I believe that we do need an auto-creation mode and also a
> way to optimize the transfer to avoid this (since it results in a lot of
> extra directory checks that can slow things down when we know that they
> aren't needed).  Which one is the default is the current question.  I'm
> currently leaning toward going back to sending the implied dirs by
> default, and having an option for people to optimize the transfer (which
> would allow it to be used with the normal -R mode even when --from-files
> was not used).

I know i'm not talking about when -R is used.  I am talking
about creating implied intermediate directories without -R.
I'm talking about being able to take the output of
find -name '*.jpg' and have it create (if necessary) any
intermediate directories while maintaining the equivalency
of src and dest.  If that means also behaving as though
those directories were already in the list that would be OK
as long as -r weren't specified.

	find . -name '*.jpg' | rsync -a --files-from=- .  remote:
should when it hits
	./deltapics/031CGMUa.jpg
	./deltapics/031CGNga.jpg
	./deltapics/031CGOHa.jpg
	./deltapics/031CGPOa.jpg
	./deltapics/031CGPba.jpg
create the deltapics directory if it doesn't exist.  The
permissions and ownership should be derived from the source.
so effectively it should be as though
	./deltapics
where in the file list.  It needn't be updated if it
does exist but if easier to implement it that way i wouldn't
object.  In such a case even if -r is
allowed and specified the implied directory should not defeat
the the file list by transferring any files not in the list.

No errors, no need to do a run to find the missing
directories and add them and no need to add a filter to the
stream adding entries for directories that are missing.


> > As it is now, if somebody just does "find . -print | rsync -a
> > --files-from- ..." are they going to get repeated files because the
> > directories are listed?
> 
> Rsync would weed out the duplicates, but if -a implied -r in this
> context then the presence of the directories would cause rsync to
> recurse through all the directory content and thus make this a
> horrible thing to do.  A couple alternatives:
> 
>   find . -print | rsync -lptgoDR --files-from=- . remote:/dest
>   find . -type f -print | rsync -aR --files-from=- . remote:/dest

Assume the user is working from a file list stored in a file
and isn't using -R.  Possibly doing a pull.  Let's not force
them to go through extra hoops.  I know i've never used -R
and i suspect many other people haven't.  Try

	find $srddir -assorted-options|sed -e 's/^srcdir\///' \
		|rsync -a --files $srcdir remote:$destdir

While unlikely from the command line, not unlikely from a
script.  While fairly easy to do simple subs, adding lines
can be a pain.


-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw at pegasys.ws

		Remember Cernan and Schmitt