memory usage in rsync 3.0.3 -- how much RAM should I have to transfer 13 million files?

[Wayne Davison]

On Tue, Aug 12, 2008 at 11:46:11AM -0700, Aleksey Tsalolikhin wrote:
> How much memory will rsync use?   I didn't specificy any of the
> switches that disable incremental recursion.

It depends on your options, and possibly on the maximum number of files
in a directory.  I've seen a recursive scan use about 20MB for a huge
set of files, which is not much more memory than a typical bash or zsh
process uses.

You should make extra sure that you're not disabling incremental
recursion by specifying at least one -v option and checking that rsync
says that it is sending/receiving an incremental file list at the start
(not the older "building file list" or "receiving file list" messages).

If it is doing an incremental scan, it still processes the immediate
contents of every individual directory as a unit (and has at least one
directory of read-ahead in memory), so if you have really huge numbers
of files in your directories, that will increase the maximum memory
used.

Other options will also affect memory use, such as -H which makes rsync
search for hard-link matches over the whole of the hierarchy, and that
can really bloat things, particularly if a large percentage of the files
in the transfer have more than one link.  (Since you didn't cite your
options, I can't be more specific.)

..wayne..