Pipelining arbitrary file transfers + --{in,ex}clude oddness

Paul Makepeace samba.org at paulm.com
Fri Jul 12 18:51:34 EST 2002

I'd like to rsync a series of files at arbitrary points in a tree in one
connection and am struggling to figure out how to do this. rsync seems
AFAICS to have a fundamental src_file dst_file model, optionally with

The best I could come up with was recursing plus multiple --include with
a final --exclude "*". The idea is to avoid multiple calls to rsync and
thus TCP/session overhead costs. Is this reasonable? A better way?
(Sneaking feeling I'm missing something really obvious at 2am, despite
looking in the FAQ, man page, google and the wishlist...)

Anyway, doesn't this scheme make the server work hard building a
file list at its end?


In the course of playing with this I found something I can't explain.
On one server, this scheme seems to work fine but another says there's
nothing there. If I try to get the file directly it's reported there. In
other words, (missing final --exclude "*" for clarity)

	rsync -a rsync://server/module --include path/file # "nothing to do"
	rsync -a rsync://server/module/path/file           # shows file & path info

Here's a transcript to demonstrate this: (yes, I'm playing with apt-get
update speed-ups but perhaps ignore that for now :-)
Using: rsync  version 2.5.6cvs  protocol version 26

1. non-us seems to work fine:

$ rsync -Rzav --progress rsync://non-us.debian.org/debian-non-US/dists/testing/non-US/main/binary-i386/Packages
receiving file list ... 
6 files to consider
drwxrwsr-x        4096 2002/07/12 01:26:18 dists
lrwxrwxrwx           3 2001/02/07 13:59:15 dists/testing -> woody
drwxrwsr-x        4096 2002/07/12 19:53:44 dists/testing/non-US
drwxrwsr-x        4096 2001/01/17 23:52:49 dists/testing/non-US/main
drwxrwsr-x        4096 2002/07/12 19:52:41 dists/testing/non-US/main/binary-i386
-rw-rw-r--      127844 2002/07/11 19:52:45 dists/testing/non-US/main/binary-i386/Packages
wrote 133 bytes  read 200 bytes  666.00 bytes/sec
total size is 127847  speedup is 383.92

2. security ditto:

$ rsync -Rzav --progress rsync://security.debian.org/debian-security/dists/testing/updates/main/binary-i386/Packages
receiving file list ... 
6 files to consider
drwxrwsr-x        4096 2002/05/01 13:14:24 dists
lrwxrwxrwx           5 2002/05/10 02:50:17 dists/testing -> woody
drwxrwsr-x        4096 2002/07/12 21:40:06 dists/testing/updates
drwxrwsr-x        4096 2002/05/01 13:14:13 dists/testing/updates/main
drwxrwsr-x        4096 2002/07/12 21:40:00 dists/testing/updates/main/binary-i386
-rw-rw-r--       88472 2002/07/12 21:22:39 dists/testing/updates/main/binary-i386/Packages
wrote 137 bytes  read 199 bytes  224.00 bytes/sec
total size is 88477  speedup is 263.32

3. back to non-us using the --include path/file --exclude "*" hack.
   All's well:

$ rsync -Rzav --progress rsync://non-us.debian.org/debian-non-US/ --include dists/testing/non-US/main/binary-i386/Packages --exclude "*"
receiving file list ... 
1 file to consider
-rw-rw-r--      127844 2002/07/11 19:52:45 dists/testing/non-US/main/binary-i386/Packages
wrote 144 bytes  read 118 bytes  524.00 bytes/sec
total size is 127844  speedup is 487.95

4. this fails on security however:

$ rsync -Rzav --progress rsync://security.debian.org/debian-security/ --include dists/testing/updates/main/binary-i386/Packages --exclude "*"
receiving file list ... 
0 files to consider
client: nothing to do: perhaps you need to specify some filenames or the --recursive option?

Case 4 seems to apply for http.us.debian.org too. I've only seen non-us


Paul Makepeace ....................................... http://paulm.com/

"If I thought it would help, then you can place a hammer in a cake."
   -- http://paulm.com/toys/surrealism/

More information about the rsync mailing list