Pipelining arbitrary file transfers + --{in,ex}clude oddness
Dave Dykstra
dwd at bell-labs.com
Mon Jul 22 11:11:35 EST 2002
Yes, doing explicit includes followed by exclude '*' is what people do.
Because you exclude '*' and the rsync algorithm is recursive, you need to
explicitly include all parent directories of the files you want to copy
as well.
We have discussed making a --files-from option with which people can
explicitly list only the files they want to copy, and I had promised to
write it, but I haven't been able to find the time.
- Dave Dykstra
On Sat, Jul 13, 2002 at 02:43:12AM +0100, Paul Makepeace wrote:
> I'd like to rsync a series of files at arbitrary points in a tree in one
> connection and am struggling to figure out how to do this. rsync seems
> AFAICS to have a fundamental src_file dst_file model, optionally with
> recursion.
>
> The best I could come up with was recursing plus multiple --include with
> a final --exclude "*". The idea is to avoid multiple calls to rsync and
> thus TCP/session overhead costs. Is this reasonable? A better way?
> (Sneaking feeling I'm missing something really obvious at 2am, despite
> looking in the FAQ, man page, google and the wishlist...)
>
> Anyway, doesn't this scheme make the server work hard building a
> file list at its end?
>
> **
>
> In the course of playing with this I found something I can't explain.
> On one server, this scheme seems to work fine but another says there's
> nothing there. If I try to get the file directly it's reported there. In
> other words, (missing final --exclude "*" for clarity)
>
> rsync -a rsync://server/module --include path/file # "nothing to do"
> rsync -a rsync://server/module/path/file # shows file & path info
>
> Here's a transcript to demonstrate this: (yes, I'm playing with apt-get
> update speed-ups but perhaps ignore that for now :-)
> Using: rsync version 2.5.6cvs protocol version 26
>
> 1. non-us seems to work fine:
>
> $ rsync -Rzav --progress rsync://non-us.debian.org/debian-non-US/dists/testing/non-US/main/binary-i386/Packages
> receiving file list ...
> 6 files to consider
> drwxrwsr-x 4096 2002/07/12 01:26:18 dists
> lrwxrwxrwx 3 2001/02/07 13:59:15 dists/testing -> woody
> drwxrwsr-x 4096 2002/07/12 19:53:44 dists/testing/non-US
> drwxrwsr-x 4096 2001/01/17 23:52:49 dists/testing/non-US/main
> drwxrwsr-x 4096 2002/07/12 19:52:41 dists/testing/non-US/main/binary-i386
> -rw-rw-r-- 127844 2002/07/11 19:52:45 dists/testing/non-US/main/binary-i386/Packages
> wrote 133 bytes read 200 bytes 666.00 bytes/sec
> total size is 127847 speedup is 383.92
>
> 2. security ditto:
>
> $ rsync -Rzav --progress rsync://security.debian.org/debian-security/dists/testing/updates/main/binary-i386/Packages
> receiving file list ...
> 6 files to consider
> drwxrwsr-x 4096 2002/05/01 13:14:24 dists
> lrwxrwxrwx 5 2002/05/10 02:50:17 dists/testing -> woody
> drwxrwsr-x 4096 2002/07/12 21:40:06 dists/testing/updates
> drwxrwsr-x 4096 2002/05/01 13:14:13 dists/testing/updates/main
> drwxrwsr-x 4096 2002/07/12 21:40:00 dists/testing/updates/main/binary-i386
> -rw-rw-r-- 88472 2002/07/12 21:22:39 dists/testing/updates/main/binary-i386/Packages
> wrote 137 bytes read 199 bytes 224.00 bytes/sec
> total size is 88477 speedup is 263.32
>
> 3. back to non-us using the --include path/file --exclude "*" hack.
> All's well:
>
> $ rsync -Rzav --progress rsync://non-us.debian.org/debian-non-US/ --include dists/testing/non-US/main/binary-i386/Packages --exclude "*"
> receiving file list ...
> 1 file to consider
> -rw-rw-r-- 127844 2002/07/11 19:52:45 dists/testing/non-US/main/binary-i386/Packages
> wrote 144 bytes read 118 bytes 524.00 bytes/sec
> total size is 127844 speedup is 487.95
> asterix:/home/paulm#
>
> 4. this fails on security however:
>
> $ rsync -Rzav --progress rsync://security.debian.org/debian-security/ --include dists/testing/updates/main/binary-i386/Packages --exclude "*"
> receiving file list ...
> 0 files to consider
> client: nothing to do: perhaps you need to specify some filenames or the --recursive option?
>
> Case 4 seems to apply for http.us.debian.org too. I've only seen non-us
> work.
>
> Cheers,
> Paul
>
> --
> Paul Makepeace ....................................... http://paulm.com/
>
> "If I thought it would help, then you can place a hammer in a cake."
> -- http://paulm.com/toys/surrealism/
>
> --
> To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
More information about the rsync
mailing list