rsync 2.6.0 - suspected memory leak bug

[jw schultz]

On Wed, Jan 21, 2004 at 03:35:37PM +0000, Kelly Garrett wrote:
> Wayne Davison <wayned <at> samba.org> writes:
> 
> > 
> > On Tue, Jan 20, 2004 at 06:16:34PM -0800, jw schultz wrote:
> > > As far as i can tell Under inetd each connection should get
> > > independant rsync process(es) which all exit so there would be no
> > > rsync processes running unless there is an active connection.
> > 
> > Yes, that's my understanding as well.  He also said that the receiving
> > side is initiating the connection (and having the problem), so I'm
> > assuming that he's not complaining about the daemon side.
> > 
> > One other thing I thought of:  Kelly, check for lingering rsync
> > processe(s) in the process table.  That's the only way I know that
> > memory could continue to be allocated at the end of a transfer:  if
> > one of the processes didn't actually exit.
> > 
> > ..wayne..
> 
> There are no residual rsync processes running after the transfer.  I think you 
> hit it on the head when you indicated that Linux would use available mem for 
> cacheing that is messing up.  Rsync seems to be exiting correctly, and I have 
> tried it with several different versions of rsync.  The way that the kernel is 
> currently compiled/configured seems not to be releasing the diskcache memory 
> back to other processes when needed, and the machine crashes once all available 
> RAM and swap has been consumed by the cache during the rsync transfer.  So, I 
> do not beleive it is rsync's "fault" -- rsync merely exposed a flaw in the OS 
> due to the massive size of the transfers we are doing with it.  My appologies 
> to the rsync team, it is an excellent tool and had I not been impressed with it 
> I would not have bothered tracking down the problem.
> 
> Does anyone know how to build a version of the kernel that either does no disk 
> cacheing (we have very fast RAID processors and SCSI disks on the machine) or 
> limit the amount of cache that the system will allocate for disk?
> 
> Any help would be appreciated.

This is actually a fairly common problem.  From your
description it sounds a bit more like the dcache than the
page cache but either way it has appeared frequently on the
Linux kernel mailing list (lkml).  You might want to search
the lkml archives for things like "paging broken", "out of
memory", "updatedb" and "swapiness".  I seem to recall there
being some tunable as well as patches.  The 2.4.20 kernel
is a bit old as far as the lkml guys are concerned but i'll
bet there is something you can do short of upgrading to
a vanilla kernel.

-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw at pegasys.ws

		Remember Cernan and Schmitt