rsync 2.6.0 - suspected memory leak bug

[Jim Salter]

IANALG (I Am Not A Linux Guy - FreeBSD is my *nix of choice), but could 
this possibly be related to the mem:remap Linux kernel bug that was 
recently discovered?

Jim Salter


> Thanks Wayne,
> 
> This just may explain the behavior.  I noticed that when I deleted the
> files that were transferred the memory was freed back up.  Thanks for
> your help, I really appreciate it - I am not a Linux kernel guru so some
> of the low-level subtleties of the OS are lost on me =8^).  
> 
> I am starting to think there may be a kernel bug (or, maybe more likely,
> a misconfiguration), though.  If it is the disk cache that is eating up
> this memory it does not seem to be freeing up for other processes when
> needed.  I will eventually get a kernel panic on the machine when RAM
> and swap have been totally consumed during the rsync transfer.  If this
> is the case, I may be out of luck, as Red Hat has retired RH8 so there
> will be no bug fixes on it - and I can't get 800GB+ of RAM into the
> machine to keep it running past the transfers.  Rebooting a busy
> production machine every hour or so (before the panic can happen) is
> just not practical (geesh, sounds like I am running Windows).  The
> application we are running will not run with the newer versions of RH
> yet, so I may have to go to a Sun solution.
> 
> Cheers
> 
> Kelly
> 
> 
> -----Original Message-----
> From: Wayne Davison [mailto:wayned at samba.org] 
> Sent: Tuesday, January 20, 2004 9:09 PM
> To: Garrett, Kelly
> Cc: rsync at lists.samba.org
> Subject: Re: rsync 2.6.0 - suspected memory leak bug
> 
> On Tue, Jan 20, 2004 at 02:45:47PM -0700, Garrett, Kelly wrote:
> 
>>After every rsync transfer there is a large amount of memory that is
>>not freed up.
> 
> 
> Your report sounds like you're not talking about process size, but a
> free-memory report from something like "top".  If so, Linux uses unused
> memory as disk cache, so the more disk I/O that happens, the less free
> memory you'll see on your system.  This isn't a bad thing, though, as
> this disk-cache memory will get used for process memory as needed.  Also
> keep in mind that once a process terminates, there's no way it can
> continue to hold memory (unless there's a bug in the kernel).
> 
> If you meant something else, please explain what you're measuring.  In
> my tests rsync's memory size stays steady throughout the transfer (once
> the file list has been built).  Shared memory between the forked
> processes on the receiving side does slowly become unshared, but that
> happened in prior rsync versions too (and we've got an internal change
> in CVS that should make this better for the next release).
> 
> ..wayne..