Pipelining arbitrary file transfers + --{in,ex}clude oddness

Dave Dykstra dwd at bell-labs.com
Mon Jul 22 11:11:35 EST 2002


Yes, doing explicit includes followed by exclude '*' is what people do.
Because you exclude '*' and the rsync algorithm is recursive, you need to
explicitly include all parent directories of the files you want to copy
as well.

We have discussed making a --files-from option with which people can
explicitly list only the files they want to copy, and I had promised to
write it, but I haven't been able to find the time.

- Dave Dykstra


On Sat, Jul 13, 2002 at 02:43:12AM +0100, Paul Makepeace wrote:
> I'd like to rsync a series of files at arbitrary points in a tree in one
> connection and am struggling to figure out how to do this. rsync seems
> AFAICS to have a fundamental src_file dst_file model, optionally with
> recursion.
> 
> The best I could come up with was recursing plus multiple --include with
> a final --exclude "*". The idea is to avoid multiple calls to rsync and
> thus TCP/session overhead costs. Is this reasonable? A better way?
> (Sneaking feeling I'm missing something really obvious at 2am, despite
> looking in the FAQ, man page, google and the wishlist...)
> 
> Anyway, doesn't this scheme make the server work hard building a
> file list at its end?
> 
> **
> 
> In the course of playing with this I found something I can't explain.
> On one server, this scheme seems to work fine but another says there's
> nothing there. If I try to get the file directly it's reported there. In
> other words, (missing final --exclude "*" for clarity)
> 
> 	rsync -a rsync://server/module --include path/file # "nothing to do"
> 	rsync -a rsync://server/module/path/file           # shows file & path info
> 
> Here's a transcript to demonstrate this: (yes, I'm playing with apt-get
> update speed-ups but perhaps ignore that for now :-)
> Using: rsync  version 2.5.6cvs  protocol version 26
> 
> 1. non-us seems to work fine:
> 
> $ rsync -Rzav --progress rsync://non-us.debian.org/debian-non-US/dists/testing/non-US/main/binary-i386/Packages
> receiving file list ... 
> 6 files to consider
> drwxrwsr-x        4096 2002/07/12 01:26:18 dists
> lrwxrwxrwx           3 2001/02/07 13:59:15 dists/testing -> woody
> drwxrwsr-x        4096 2002/07/12 19:53:44 dists/testing/non-US
> drwxrwsr-x        4096 2001/01/17 23:52:49 dists/testing/non-US/main
> drwxrwsr-x        4096 2002/07/12 19:52:41 dists/testing/non-US/main/binary-i386
> -rw-rw-r--      127844 2002/07/11 19:52:45 dists/testing/non-US/main/binary-i386/Packages
> wrote 133 bytes  read 200 bytes  666.00 bytes/sec
> total size is 127847  speedup is 383.92
> 
> 2. security ditto:
> 
> $ rsync -Rzav --progress rsync://security.debian.org/debian-security/dists/testing/updates/main/binary-i386/Packages
> receiving file list ... 
> 6 files to consider
> drwxrwsr-x        4096 2002/05/01 13:14:24 dists
> lrwxrwxrwx           5 2002/05/10 02:50:17 dists/testing -> woody
> drwxrwsr-x        4096 2002/07/12 21:40:06 dists/testing/updates
> drwxrwsr-x        4096 2002/05/01 13:14:13 dists/testing/updates/main
> drwxrwsr-x        4096 2002/07/12 21:40:00 dists/testing/updates/main/binary-i386
> -rw-rw-r--       88472 2002/07/12 21:22:39 dists/testing/updates/main/binary-i386/Packages
> wrote 137 bytes  read 199 bytes  224.00 bytes/sec
> total size is 88477  speedup is 263.32
> 
> 3. back to non-us using the --include path/file --exclude "*" hack.
>    All's well:
> 
> $ rsync -Rzav --progress rsync://non-us.debian.org/debian-non-US/ --include dists/testing/non-US/main/binary-i386/Packages --exclude "*"
> receiving file list ... 
> 1 file to consider
> -rw-rw-r--      127844 2002/07/11 19:52:45 dists/testing/non-US/main/binary-i386/Packages
> wrote 144 bytes  read 118 bytes  524.00 bytes/sec
> total size is 127844  speedup is 487.95
> asterix:/home/paulm#
> 
> 4. this fails on security however:
> 
> $ rsync -Rzav --progress rsync://security.debian.org/debian-security/ --include dists/testing/updates/main/binary-i386/Packages --exclude "*"
> receiving file list ... 
> 0 files to consider
> client: nothing to do: perhaps you need to specify some filenames or the --recursive option?
> 
> Case 4 seems to apply for http.us.debian.org too. I've only seen non-us
> work.
> 
> Cheers,
> Paul
> 
> -- 
> Paul Makepeace ....................................... http://paulm.com/
> 
> "If I thought it would help, then you can place a hammer in a cake."
>    -- http://paulm.com/toys/surrealism/
> 
> -- 
> To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html




More information about the rsync mailing list