rsync many processes and slow backup
Fabian Cenedese
Cenedese at indel.ch
Fri Jul 12 09:30:46 UTC 2019
At 10:56 12.07.2019, Hannes Hutmacher via rsync wrote:
>Hi all! :-)
>
>I have a small rsync script to sync my data to a usb-disk. It works fine, when I start it in console. I get 3 rsync processes (look in top) and the backup takes ~25 min. But, when I add the script in cron to start it at 1am at night it takes 7 - 9 hours and I see up to 180 processes. When I look in top I see a hight load of 60 - 80 and 40 - 60 waits. Why? Can someone explain why it takes so long when it starts with cron?
>
>This is my rsync command: rsync -azc --delete "$QUELLORDNER" "$ZIELORDNER"
>This is the entry in cron (crontab -e): * 2 * * * /root/backupscript/backup.sh
I'd say this starts rsync on every minute between 2:00 and 3:00.
You should change that to e.g. 0 2 ...
>Data to sync: 18 Gb, 185.000 files.
>
>When I look in the log files I see errors like this:
>
>rsync: rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c" -> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2)
That's probably because several rsync instances are stomping
on each other's feet.
bye Fabi
More information about the rsync
mailing list