memory requirements was RE: out of memory in build_hash_table

[David Bolen]

Granzow, Doug (NCI) [granzowd at mail.nih.gov] writes:

> From what I've observed by running top while rsync is running, its memory
> usage appears to grow gradually, not exponentially.

The exponential portion of the growth is up front when rsync gathers
the file listing (it starts with room for 1000 files, then doubles
that to 2000, 4000, etc...).  So if your rsync has started
transferring at least the first file, it's already done whatever
exponential growth it's going to do.  After that yes, it's far more
gradual and should I think settle down since most of the rest of the
memory allocation is on a per-file basis and not saved once the
individual file is done.

> I saw someone on this list recently mentioned changing the block size.
The
> rsync man page indicates this defaults to 700 (bytes?).  Would a larger
> block size reduce memory usage, since there will be fewer (but larger)
> blocks of data, and therefore fewer checksums to store?

Yep, although I think that is a reasonably small amount of memory
(something like 8 bytes per block to hold the checksums) and only
holds the checksums for a single file at a time.  But in addition to
memory for larger files this can also improve performance because
there's less computation to be done as well as less block matching.
The downside is potentially more data transferred.

> You suggested setting ARENA_SIZE to 0... I guess this would be done like
> this?
> 
> % ARENA_SIZE=0 ./configure

I don't know if the configure script looks in the environment or not,
but my guess would be no.  (Took a quick peek and it doesn't look like
that's something munged by configure at all).

If you wanted to try it, I'd just edit rsync.h and comment out it's
current definition in favor of one defining it to 0 - e.g.:

    /* #define ARENA_SIZE      (32 * 1024) */
    #define ARENA_SIZE 0

The arena handling seems to be reasonably tight, so it's probably a
long shot in any event.

-- David

/-----------------------------------------------------------------------\
 \               David Bolen            \   E-mail: db3l at fitlinxx.com  /
  |             FitLinxx, Inc.            \  Phone: (203) 708-5192    |
 /  860 Canal Street, Stamford, CT  06902   \  Fax: (203) 316-5150     \
\-----------------------------------------------------------------------/