stalling during delta processing

[Craig Barratt]

"Wallace Matthews" writes:

> I copy the 29 Gig full backup back into fedor//test/Kibbutz and issue
> the command "time rsync -avv --rsh=rsh --stats --block-size=181272
> /test/Kibbutz/Kbup_1 fedor://test/Kibbutz" and it CRAWLS during delta
> generation/transmittal at about 1 Megabyte per second.
>
> I have repeated the experiment 3 times; same result each time.
>
> The only thing that is different is --block-size= option. First,
> time it isnt specified and I get a predictable answer. Second
> time, I give it a block size that is about 1/2 of square root of
> (29 Gig) and that is ok. But, explicitly give it something that
> is approximately the square root of the 29 Gig and it CRAWLS.
>
> When I cancel the command, the real time is 86 minutes and the
> user time is 84 minutes. This is similar to the issue I reported
> on Friday that Chris suggested I remove the --write-batch= option
> and that seemed to fix the CRAWL.

If I understand the code correctly, map_ptr() in filio.c maintains
a sliding window of data in memory.  The window starts 64K prior
to the desired offset, and the window length is 256K.  So your
block-size of 181272 occupies most of the balance of the window.

Each time you hit the end of the window the data is memmoved
and the balance needed is read.  With such a large block size
there will be a lot of memmoves and small reads.

I doubt this issue explains the dramatic reduction in speed, but
it might be a factor.  Perhaps there is a bug with large block
sizes?

And, yes, your observation about the number of matching blocks
needs to be explored.

Craig