Rsync memory usage seems a lot higher than 100bytes/file

Surer Dink surerlistmail at gmail.com
Thu Jun 15 15:50:52 GMT 2006


All,
  I have read the lists, I have read the faq - yes, rsync uses a lot
of memory.  That said - I need to use rsync in an environment with a
lot of files, and I need to use -H and --delete options.  The faq says
about 100 bytes per file - at the moment I am looking at 8700000
files, which should translate to about 800MB - but it does not.  In
fact, running rsync not only consumes all of the ram, but also all of
the swap in the machine - 1.5GB+1.7GB.  Is this because I am using -H
and --delete?  What is the per-file overhead when both of these
options are used?
  What do others do when faced with this problem?  Although the FAQ
confirms that rsync uses much ram, it does not actually provide an
answer or a workaround.  I have tried doing partial rsyncs (one half
of the directories in one operation, one half in the other) however
this fails to preserve the hardlinks - and I need for these hardlinks
to remain.  I would really appreciate some hints as to how I can work
arround this issue, or at least figure out why so much more ram than
was expected is being used.
  Thanks!  (If it matters, the operation is being run on two disks
connected to a FreeBSD 6.1 machine).


More information about the rsync mailing list