rsync many processes and slow backup

[Henri Shustak]

I think others have answered your questions. You may want to implement some sort of locking system in the script so that you can only run a single instance of your script at. If you are looking for a backup tool which leverages rsync you may want to take a look at LBackup : http://www.lbackup.org

There is even a pre-script you can enable (in the example resources) so that the script can run a maximum of once a day / once a month etc (even if you start it many times from cron.

Hope you have it all working now :)

Henri


>> Hi all! :-)
>> 
>> I have a small rsync script to sync my data to a usb-disk. It works
>> fine, when I start it in console. I get 3 rsync processes (look in
>> top) and the backup takes ~25 min. But, when I add the script in
>> cron to start it at 1am at night it takes 7 - 9 hours and I see up to
>> 180 processes. When I look in top I see a hight load of 60 - 80 and 40
>> - 60 waits. Why? Can someone explain why it takes so long when it
>> starts with cron?
>> 
>> This is my rsync command:  rsync -azc --delete "$QUELLORDNER"
>> "$ZIELORDNER" 
>> This is the entry in cron (crontab -e): * 2 * * *
>> /root/backupscript/backup.sh
>> Data to sync: 18 Gb, 185.000 files.
>> 
>> When I look in the log files I see errors like this: 
>> 
>> rsync:
>> rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c"
>> -> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) 
>> 
>> rsync error: some files/attrs were not transferred (see previousrsync
>> error:
>> 
>> some files/attrs were not transferred (see previous errors) (code 23) at
>> 
>> main.c(1196) [sender=3.1.2]
>> 
>> directory (2)
>> 
>> 
>> 
>> Can you help me to solve this problem?
>> 
>> 
>> 
>> regards,
>> 
>> Hannes Hutmacher
>> 
>> 
> A couple of questions: (I am not an expert user.)
> 
> Does your manual job run as the same user (presumably root) as does your
> cron job?
> 
> Are you backing up any temporary files that you might be better off
> excluding?
> 
> Is anything else running at the same time as your cron job which may be
> creating and destroying files in the backup source or target? (E.g. Your
> firewall?)
> 
> I'm not very familiar with fuzzy searching, but using that in places
> like /var seems odd.
> 
> One of the constant refrains on this list is that using checksums is
> almost never a good idea. In combination with fuzzy, it seems even more
> tenuous.
> 
> Joe
> 
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html