rsync memory usage

[tim.conway at philips.com]

Lutz:  This has been a frequent topic of discussion.  Wayne Davidson (and 
probably others) is working on a different way, but currently, this method 
is integral to rsync, and to change it will be a major revision.  For my 
application, I actually had to write my own syncronization program that 
breaks down the comparison and propogation processes to thinner streams, 
but my procedure acts like rsync with -W, and with no capability to do -c.
You may have to do what many others do - break your trees into branches 
and burn them one at a time.

Tim Conway
tim.conway at philips.com
303.682.4917 office, 3039210301 cell
Philips Semiconductor - Longmont TC
1880 Industrial Circle, Suite D
Longmont, CO 80501
Available via SameTime Connect within Philips, n9hmg on AIM
perl -e 'print pack(nnnnnnnnnnnn, 
19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970), 
".\n" '
"There are some who call me.... Tim?"




Lutz Pressler <Lutz.Pressler at SerNet.DE>
Sent by: rsync-admin at lists.samba.org
07/17/2002 03:27 AM

 
        To:     <rsync at lists.samba.org>
        cc:     (bcc: Tim Conway/LMT/SC/PHILIPS)
        Subject:        rsync memory usage
        Classification: 



Hi,

we are using rsync to mirror large trees (>> 50 GB, >> 2 mio files)
offsite (rsync -e ssh with forced ssh command on the remote side).
The main problem occuring is memory usage (especially as the remote
system has no swap space configured for security reasons):

It seems that the rsync processes' RSS on both sides increases with
the number of files in the tree. Rsyncing the whole tree at once is
not possible anymore, as the remote system gets out of memory.

At the moment, we're rsyncing (sub)subdirectories seperately to not
reach the memory limit. But this is not really feasible anymore,
because of dynamic spread of content and limitations by the ssh
forced command method.

Why is rsync holding data for the whole tree in memory?

Any idea for a solution to this problem?

Thank you,
  Lutz Pressler

P.S.: rsync versions 2.5.5 on the remote and 2.4.6 on the local side.
      Ask for exact command lines if necessary.


-- 
  _              |  Lutz Pressler          |  Tel: ++49-551-3700002
 |_     |\ |     |  Service Network GmbH   |  FAX: ++49-551-3700009
 ._|ER  | \|ET   |  Bahnhofsallee 1b       |   mailto:lp at SerNet.DE
Service Network  |  D-37081 Goettingen     |  http://www.SerNet.DE/


-- 
To unsubscribe or change options: 
http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html