High memory usage - any way around it other than splitting jobs?

[Perry Hutchison]

Andy Smith via rsync <andy at lists.samba.org> wrote:

> I have a virtual machine with 2G of memory. On this VM there is a
> directory tree with 33.3 million files in it. When attempting to
> rsync (rsync -PSHav --delete /source /dest) this tree from one
> directory to another on the same host, rsync uses all the memory and
> is killed by oom-killer.
>
> This host is Debian oldstable so has
>
> $ rsync --version
> rsync  version 3.1.2  protocol version 31

Since this is all taking place on a single VM (thus there is no
network involved), it's possible that rsync is not the best tool
for the job.  Had you considered something like:

$ ( cd /source && find . -depth -print0 | cpio -p -0l /dest ) && rm -rf /source

One advantage of "cpio -p -l" is that it avoids copying any of the
files -- it just makes a new directory tree containing hardlinks to
the original files.

(I am guessing that, since this is a VM, it is likely to have been
set up with a single, large filesystem on a single (virtual) drive
rather than the older approach of creating multiple partitions --
so that /source and /dest are in the same filesystem.)

For that matter, what about:

# rm -rf /dest
# mv /source /dest

i.e. just rename the source tree?