Rsync: Re: patch to enable faster mirroring of large filesystems

Dave Dykstra dwd at
Thu Nov 29 08:36:33 EST 2001

Rsync list: Alberto and I have done a couple more exchanges by private
email, and we found that he wasn't turning on my include/exclude
optimization in his test because he had an "exclude" directive in
rsyncd.conf.  He has now removed that and run the test again.  His very
interesting results are below with my comments.

Note that his case is rather pathological because he's got over a million
files in only 400 directories, so he must have an average of over 2500
files per directory, which are very large directories.  He's got about 65%
of the files explicitly listed in his --include-from file.

On Wed, Nov 28, 2001 at 03:18:24PM -0500, Alberto Accomazzi wrote:
> Both machines are SUN ultra 80s (2x450 UII, 1GB RAM), on a rather busy LAN,
> so take results with a grain of salt:
> # syncronization of 1.1M files in 400 directories (135,000 to be updated):
> > date ; rsync-2.3.2 -avvzn rsync:// . ; date
> Wed Nov 28 10:01:15 EST 2001
> receiving file list at Wed Nov 28 10:01:17 2001
> done receiving file list at Wed Nov 28 10:27:17 2001
> [ ...list of approx 135,000 files to be updated or 2.4 MB... ]
> wrote 539699 bytes  read 17803469 bytes  11046.77 bytes/sec
> total size is 1137025227  speedup is 61.99
> Wed Nov 28 10:28:55 EST 2001
> # syncronization of 722,941 files (13MB) in bib.list from the same directory
> > date ; rsync-2.3.2 -avvzn --include-from bib.list --exclude '*' rsync:// . ; date
> Wed Nov 28 10:53:03 EST 2001
> sending exclude list at Wed Nov 28 11:56:22 2001
> done sending exclude list at Wed Nov 28 14:13:48 2001
> receiving file list at Wed Nov 28 14:13:48 2001
> (using include-only optimization) done receiving file list at Wed Nov 28 14:18:59 2001
> [ ...list of approx 3,200 files to be updated or 58 KB... ]
> wrote 16640660 bytes  read 10001913 bytes  2143.15 bytes/sec
> total size is 755673302  speedup is 28.36
> Wed Nov 28 14:20:15 EST 2001

Note the difference in total bytes written; presumably it was the exclude

> The astonishing thing here is the time spent by the client in fiddling and
> sending the exclude list!  Just over 1hr to create the list in memory and 
> more than 2hrs to send it.  When I trussed the process during the exclude
> list sending time this is what I saw for every file:
> [...]
> write(3, " +  ", 2)                             = 2
> poll(0xFFBED858, 1, 60000)                      = 1
> write(3, " J 9 0 / J 9 0 - 0 5 8 6".., 17)      = 17
> poll(0xFFBED850, 1, 60000)                      = 1
> write(3, "13\0\0\0", 4)                         = 4
> [...]
> so it looks like sending the exclude list is quite inefficient and therefore
> --file-from should definitely not use this code to do the same thing.

Indeed, it definitely should be doing buffering!  It looks like there's a
function io_start_buffering() that should be called.  I don't know why it
isn't called until later, though, and there may be a good reason.  It's
getting called in send_file_list() and in do_recv(), both of which are
called from client_run() in main.c after send_exclude_list().  Could you
play with calling it before send_exclude_list()?  You're sending the list
from the receiver to the sender so you're in the second half client_run(),
the do_recv() part.  You may possibly need to call io_flush() more often,
although I don't think so.  Could that be the reason why io_start_buffering()
wasn't turned on earlier?  Looks like buffering can be disabled with
io_end_buffering() if you need to.

> Also I'm sure that the 1hr spent building the exclude list can be 
> greatly reduced by just slurping in the file list in memory.

Yes, I think that 1hr can be completely bypassed by reading the
--files-from file directly inside the send_exclude_list() function
and bypassing all the work done by make_exclude_list() to generate the
in-memory representation of the exclude patterns.  I sure am glad you
ran this test because otherwise I probably wouldn't have thought of
doing that.

Hmm, wait, the remote side would still be building the in-memory exclude
pattern representations.  I guess that needs a short-circuit too.

> I guess the good news is how quickly the results came back from the server,
> which is where your optimization kicks in. 

Yes, that only took 5 minutes!

> I've started one last test that
> won't trigger the optimization out of curiosity, although these numbers
> clearly show that most of the gain can be had by bypassing the
> include/exclude dance on the client side.

I expect the 5 minutes part will go up significantly and the rest will
stay the same.  I'd like to know by how much.

- Dave Dykstra

More information about the rsync mailing list