sync performance falls off a cliff

[Leen Besselink]

Mike Connell wrote:
> Hi,
>  

Hi again Mike,

> I don't see how to reply to your post so it shows up as a reply
> on the list. So I guess I'll just send email directly to you.
>  

You just e-mail rsync at lists.samba.org instead of me. :-)

> Today I've been watching the production 2.6.8 rsync off and on and no it
> isn't swapping. Used "vmstat" and "top" both on the source and
> the destination. Each shows 0 for si and so.
>  
> With iostat -xn 5, I do see that first disk utilization on the source
> hits 95% while the file list is being received. After rsync says done
> (with the file list), then the destination hits 95% disk utilization.
>  
> This is not good. As it takes more and more time, it will be pegging
> our servers.
>  

Maybe reading all the files from disk fills up the 'file-caching'
(the memory used to prevent reading from disk).

If I do some quick calculation (and my math isn't wrong), then
375.000 * 16k almost fits in 4GB of memory and 425.000 definitly
does not.

Thus it needs to go back to disk and read from there.

> So I used your good advice and downloaded and built rsync 3.0.6.
> (Couldn't find any packages available).
>  
> I now see that the new rsync says "receiving incremental file list".
> What does this

It just means it's supposed to use less memory, because it doesn't need
to keep the whole filelist in memory. Although I suspect other tradeoffs
might be made.

> do? Sounds good. Have only verified that the new rsync seems to work
> in a test capacity. Will move it into production soon to see how it does.
>  

If what you mentioned is how I think it is, then I doubt it will help
much or maybe just for a while.

I don't know what kernel you have (and io-scheduler you are using), but I
do know their is also a 'ionice' command (in Debian-based distributions
it's part of the util-linux-package) which can be prepended to running
the rsync command which is meant to set priorities between processes
for reading and writing to/from disk.

It will possible slow down rsync even more, but atleast it wouldn't slow
down the other processes on the server.

It will still kill the file-cache though, so in that way it could still
slow down other processes.

If it's the file-cache plugging in extra memory would solve the problem
for a certain while. I don't know if you'd need to be running 64-bit for
that though (depends on the machine and CentOS).

If it's failover setup and it turns out that their is no easy solution
with rsync, maybe something at the block device level would be more
appropriate like:

http://www.drbd.org/
or
http://www.centos.org/docs/5/html/5.1/Global_Network_Block_Device/ch-gnbd.html

But it's not something I've used before.

> Thanks,
>  
> Mike
>