rsync 2.6.0 - suspected memory leak bug

[Garrett, Kelly]

Thanks Wayne,

This just may explain the behavior.  I noticed that when I deleted the
files that were transferred the memory was freed back up.  Thanks for
your help, I really appreciate it - I am not a Linux kernel guru so some
of the low-level subtleties of the OS are lost on me =8^).  

I am starting to think there may be a kernel bug (or, maybe more likely,
a misconfiguration), though.  If it is the disk cache that is eating up
this memory it does not seem to be freeing up for other processes when
needed.  I will eventually get a kernel panic on the machine when RAM
and swap have been totally consumed during the rsync transfer.  If this
is the case, I may be out of luck, as Red Hat has retired RH8 so there
will be no bug fixes on it - and I can't get 800GB+ of RAM into the
machine to keep it running past the transfers.  Rebooting a busy
production machine every hour or so (before the panic can happen) is
just not practical (geesh, sounds like I am running Windows).  The
application we are running will not run with the newer versions of RH
yet, so I may have to go to a Sun solution.

Cheers

Kelly


-----Original Message-----
From: Wayne Davison [mailto:wayned at samba.org] 
Sent: Tuesday, January 20, 2004 9:09 PM
To: Garrett, Kelly
Cc: rsync at lists.samba.org
Subject: Re: rsync 2.6.0 - suspected memory leak bug

On Tue, Jan 20, 2004 at 02:45:47PM -0700, Garrett, Kelly wrote:
> After every rsync transfer there is a large amount of memory that is
> not freed up.

Your report sounds like you're not talking about process size, but a
free-memory report from something like "top".  If so, Linux uses unused
memory as disk cache, so the more disk I/O that happens, the less free
memory you'll see on your system.  This isn't a bad thing, though, as
this disk-cache memory will get used for process memory as needed.  Also
keep in mind that once a process terminates, there's no way it can
continue to hold memory (unless there's a bug in the kernel).

If you meant something else, please explain what you're measuring.  In
my tests rsync's memory size stays steady throughout the transfer (once
the file list has been built).  Shared memory between the forked
processes on the receiving side does slowly become unshared, but that
happened in prior rsync versions too (and we've got an internal change
in CVS that should make this better for the next release).

..wayne..