measuring 2.6.1 improvements

[John Van Essen]

On Tue, 27 Apr 2004, Eric Whiting <ewhiting at amis.com> wrote:
> 
> I was watching the memory usage with a bash command line while loop
> while(true);do cat /proc/$pid/status |grep VmSize ;sleep 60;done
> 
> My last runs I changed the sleep to 5 at the end to get finer resolution on the
> memory jump. I was not expecting to see that jump -- I was also curious.

I think I can explain the jump.

When the file list is being received by the receiver, there is a
steady increase in memory usage as the data structures are allocated
for each file as the info is received.

A list of pointers to the data structures is also being built.

When that list fills up its current buffer, a new one twice the size
is allocated, the pointers are copied, and the old buffer is freed.
That freed memory is now availble to hold more file data structures.

In the beginning, the memory allocation for the file data structures
swamps the jumps due to allocation for the list.  But once the list
size reaches into the megabytes, then you see the effects of the jump.

> some more cut/paste:
> 
> dest_host:/usr/bin # while(true);do cat /proc/24624/status  |grep VmSize;sleep
> 60; done
> VmSize:     6380 kB
> VmSize:     8196 kB
> VmSize:     9236 kB
> VmSize:    10796 kB
> VmSize:    12096 kB
> VmSize:    14168 kB
> VmSize:    15468 kB
> VmSize:    16768 kB
> VmSize:    18068 kB

That was a jump of about 1200.

> VmSize:    20408 kB <- memory jump starts after this point (sender stays stable at 22M)

That was a jump of about 2400.

> VmSize:    26016 kB
> (cntl-c  to change the resolution)
> 
> dest_host:/usr/bin # while(true);do cat /proc/24624/status  |grep VmSize;sleep 5; done

That was a jump of about 4800.

> VmSize:    35884 kB
> VmSize:    35884 kB
> VmSize:    35884 kB
> VmSize:    37964 kB

That was a jump of about 9600.  The memory size stabilizes for
awhile since the previous index buffer is reused for file info data.

> VmSize:    22412 kB

And THAT was a decrease of 15500.

Since the rsync command was a 'push', the generator runs on the
destination/receiver, so maybe the 15500 is the memory that is
now shared between the generator and the receiver after the fork?

Or maybe this is the effect of some final cleanup after receiving
the file list?  This reduction may be related to this bullet item
in the NEWS file:

    * Changed hardlink info and file_struct + strings to use
      allocation pools.  This reduces memory use for large file-
      sets and permits freeing memory to the OS.  (J.W. Schultz) 

Wayne?  JW?  Any thoughts?
-- 
        John Van Essen  Univ of MN Alumnus  <vanes002 at umn.edu>