rsync daemon caching/memory usage?

[jw schultz]

On Wed, May 28, 2003 at 07:15:23PM -0500, Matt Garman wrote:
> 
> I have two computers: a Linux box which I use as my workstation
> (day-to-day use) and an OpenBSD box which I designate for server tasks:
> gateway, firewall, etc.
> 
> I use rsync to mirror many gigs worth of data from the Linux box
> (running rsync in daemon mode) to the OpenBSD machine (running an rsync
> client job nightly).
> 
> Since the amount of data and the number of files that I mirror is great,
> most of rsync's time seems to be spent computing differences in the
> files.  This appears to put quite a burden on the Linux workstation.
> The result, I belive, is that much of the Linux box's memory is consumed
> caching all this information that it is processing.
> 
> The reason I think this is because the apps I left running (e.g.  file
> browser, Mozilla, gnumeric spreadsheets, etc) need a few seconds to
> "wake up" when they are first used for the day (subsequent use meets no
> delays).  My theory is that most of my RAM is consumed in caching the
> rsync nightly job, and my actual apps are swapped out.
> 
> At one GB, I have WAY more RAM than I need for my actual work, yet it
> seems like I'm not really benefitting from having this kind of memory
> (at least initially).
> 
> Maybe I'm just making too big a deal out of this <grin> but is there any
> way to have rsync not allow itself to be cached, or maybe to force it to
> use a disk cache instead of physical RAM?  Since this jobs runs at night
> when I'm not using the computer, I don't care about degrading rsync's
> performance.

Firstly, this is not an rsync question it is a kernel question.
Your analysis is largely correct.  Linux is trying to make
the best use of available memory.  If you want to look at
extensive discussions of this issue i'd suggest you google
for interactivity and updatedb (the most famous
cache-thrasher)

This has a great deal to do with the kernel version you are
running.   Any decent VM system will exhibit this behaviour to
some degree.  At a minimum the clean pages with backing
store that have been unused for an extended period can be
expected to be reclaimed to accommodate newer file pages.
Even without rsync and updatedb this will happen some.  The
kernel guys are trying to find a balance but that is
happening in the 2.5 tree.  In 2.5 there is a swappiness
tunable and there are fctl(2) hints that we might eventually
use.


-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw at pegasys.ws

		Remember Cernan and Schmitt