[Bug 1463] New: poor performance with large block size

Wallace Matthews wmatthews at sepaton.com
Fri Jun 18 20:44:44 GMT 2004

I applied your patch and it has resolved the problem.

Thanks Craig

-----Original Message-----
From: Craig Barratt [mailto:cbarratt at users.sourceforge.net]
Sent: Thursday, June 17, 2004 11:48 PM
To: Wallace Matthews
Cc: samba-bugs at samba.org; rsync-qa at samba.org
Subject: Re: [Bug 1463] New: poor performance with large block size 

Wally writes:

> I apologize to Craig. Chris is correct.

No problem.

> I had been reading so many of Chris's highly intelligent e-mails...

Same here.

> But, the comment seems to have been right on. I have re-run the
> experiment with block sizes as small as 3000 (yes it took a long
> time to complete) all the way up to block sizes of 100000 with it
> working in reasonable times. But, when the block size approaches
> 170,000 or so, the performance degrades exponentially.
> I understand that I am testing at the very fringes of what we should
> expect rsync to do. File sizes of 25Gig and 55Gig are beyond what was
> originally envisioned (based on 64k hash buckets and a sliding window
> of 256k).

Here's a patch to try.  It basically ensures that the window is
at least 16 times the block size.  Before I'd endorse this patch
for CVS we need to make sure there aren't cases where map_ptr is
called with a much bigger length, making the 16x a bit excessive.

Perhaps I would be tempted to repeat the previous check that the
window start plus the window size doesn't exceed the file length,
although it must be at least offset + len - window_start as in
the original code.

In any case, I'd be curious if this fixes the problem.


--- rsync-2.6.2/fileio.c        Sun Jan  4 19:57:15 2004
+++ ../rsync-2.6.2/fileio.c     Thu Jun 17 19:33:26 2004
@@ -193,8 +193,8 @@
        if (window_start + window_size > map->file_size) {
                window_size = map->file_size - window_start;
-       if (offset + len > window_start + window_size) {
-               window_size = (offset+len) - window_start;
+       if (offset + 16 * len > window_start + window_size) {
+               window_size = (offset + 16 * len) - window_start;
        /* make sure we have allocated enough memory for the window */

More information about the rsync mailing list