high memory usage transferring about 27.000.000 files
valentin at vrvis.at
Tue Sep 21 13:46:34 UTC 2021
We are trying to back up a large number of small files and have issues
with high memory usage.
On both ends debian 11.0 is running with rsync version 3.2.3 protocol
Ideally we want to use the options `-a --delete --backup -e ssh` to run
this transfer and pull a backup from the target machine.
The memory usage continually rises until using all free memory on the
initiator side, about 4.5GB split across two processes. On the sender
side one process uses about half of that.
The folder to transfer contains about 11TB of data split over
~27.000.000 files in a structure with ~19.000.000 directories.
We first tried to make sure that rsync is using it's incremental
recursion mode. After reading the manual we explicitly used the
`--delete-during` option as well as trying to run without deletion or
backup mode with no noticeable improvements.
The command line options on the sender side, as far as i understand,
indicate that incremental recursion mode is requested
We also tried --omit-dir-times to reduce memory usage without success.
We desperately try not to split up the transfer for subfolders as we
would have to do so dynamically because the folder names change.
Does anyone have experience with such transfers and can help us with
reducing the memory usage?
Or is there an issue and rsync shouldn't use that much memory?
More information about the rsync