High memory usage - any way around it other than splitting jobs?
pluto at agora.rdrop.com
Thu Jun 25 16:37:07 UTC 2020
Andy Smith via rsync <andy at lists.samba.org> wrote:
> I have a virtual machine with 2G of memory. On this VM there is a
> directory tree with 33.3 million files in it. When attempting to
> rsync (rsync -PSHav --delete /source /dest) this tree from one
> directory to another on the same host, rsync uses all the memory and
> is killed by oom-killer.
> This host is Debian oldstable so has
> $ rsync --version
> rsync version 3.1.2 protocol version 31
Since this is all taking place on a single VM (thus there is no
network involved), it's possible that rsync is not the best tool
for the job. Had you considered something like:
$ ( cd /source && find . -depth -print0 | cpio -p -0l /dest ) && rm -rf /source
One advantage of "cpio -p -l" is that it avoids copying any of the
files -- it just makes a new directory tree containing hardlinks to
the original files.
(I am guessing that, since this is a VM, it is likely to have been
set up with a single, large filesystem on a single (virtual) drive
rather than the older approach of creating multiple partitions --
so that /source and /dest are in the same filesystem.)
For that matter, what about:
# rm -rf /dest
# mv /source /dest
i.e. just rename the source tree?
More information about the rsync