stalling during delta processing

[Chris Shoemaker]

On Mon, Jun 14, 2004 at 11:03:12PM -0700, Craig Barratt wrote:
> "Wallace Matthews" writes:
> 
> > I copy the 29 Gig full backup back into fedor//test/Kibbutz and issue
> > the command "time rsync -avv --rsh=rsh --stats --block-size=181272
> > /test/Kibbutz/Kbup_1 fedor://test/Kibbutz" and it CRAWLS during delta
> > generation/transmittal at about 1 Megabyte per second.
> >
> > I have repeated the experiment 3 times; same result each time.
> >
> > The only thing that is different is --block-size= option. First,
> > time it isnt specified and I get a predictable answer. Second
> > time, I give it a block size that is about 1/2 of square root of
> > (29 Gig) and that is ok. But, explicitly give it something that
> > is approximately the square root of the 29 Gig and it CRAWLS.
> >
> > When I cancel the command, the real time is 86 minutes and the
> > user time is 84 minutes. This is similar to the issue I reported
> > on Friday that Chris suggested I remove the --write-batch= option
> > and that seemed to fix the CRAWL.
> 
> If I understand the code correctly, map_ptr() in filio.c maintains
> a sliding window of data in memory.  The window starts 64K prior
> to the desired offset, and the window length is 256K.  So your
> block-size of 181272 occupies most of the balance of the window.
> 
> Each time you hit the end of the window the data is memmoved
> and the balance needed is read.  With such a large block size
> there will be a lot of memmoves and small reads.
> 
> I doubt this issue explains the dramatic reduction in speed, but
> it might be a factor.  Perhaps there is a bug with large block
> sizes?
> 

I agree that this shouldn't account for such a slow-down.  Perhaps Wally
can test this hypothesis by using a block size exactly equal to the
window size (256k?)  and one less (256k-1?) and one more (256k+1?).  



> And, yes, your observation about the number of matching blocks
> needs to be explored.

I agree.


> 
> Craig
> -- 
> To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html