out of memory in receive_file_entry rsync-2.6.2

[Eberhard Moenkeberg]

Hi,

On Tue, 10 Aug 2004, James Bagley Jr wrote:

> I've had some problems using rsync to transfer directories with more than
> 3 million files.  Here's the error message from rsync:
> 
> <snip>
> ERROR: out of memory in receive_file_entry
> rsync error: error allocating core memory buffers (code 22) at util.c(116)
> rsync: connection unexpectedly closed (63453387 bytes read so far)
> rsync error: error in rsync protocol data stream (code 12) at io.c(342)
> </snip>
> 
> I'm doing a pull on a linux system fom the HP-UX system that actually
> houses the data.  Both using rsync-2.6.2.  The one solution i've come up
> with isn't pretty, but seems to work.

How is your RAM/swap situation on the linux side?

I am rsyncing the 4 million files of ftp.gwdg.de to a backup server each 
night, one-shot, no problems. It just needs 7 or more hours...

rsync[28526] (receiver) heap statistics:
  arena:         233472   (bytes from sbrk)
  ordblks:            9   (chunks not in use)
  smblks:             3
  hblks:           1282   (chunks from mmap)
  hblkhd:     357650432   (bytes from mmap)
  allmem:     357883904   (bytes from sbrk + mmap)
  usmblks:            0
  fsmblks:           96
  uordblks:       71072   (bytes used)
  fordblks:      162400   (bytes free)
  keepcost:      135048   (bytes in releasable chunk)

Number of files: 4099094
Number of files transferred: 27160
Total file size: 1568423288614 bytes
Total transferred file size: 16916178416 bytes
Literal data: 10758972422 bytes
Matched data: 6158385953 bytes
File list size: 119674315
Total bytes written: 11990441
Total bytes read: 10885842335

wrote 11990441 bytes  read 10885842335 bytes  410271.35 bytes/sec
total size is 1568423288614  speedup is 143.92

rsync[26637] (generator) heap statistics:
  arena:         233472   (bytes from sbrk)
  ordblks:            8   (chunks not in use)
  smblks:             3
  hblks:           1285   (chunks from mmap)
  hblkhd:     357924864   (bytes from mmap)
  allmem:     358158336   (bytes from sbrk + mmap)
  usmblks:            0
  fsmblks:           96
  uordblks:       90760   (bytes used)
  fordblks:      142712   (bytes free)
  keepcost:      135048   (bytes in releasable chunk)
==== end   ======== RC=0 ============ 040810.0100 040810.0822


Cheers -e
-- 
Eberhard Moenkeberg (emoenke at gwdg.de, em at kki.org)