preventing multiple rsync processes backing up same file with cron backup

Max Arnold lwarxx at
Wed Dec 23 03:36:01 MST 2009

On Wed, Dec 23, 2009 at 09:54:05AM +0000, chris snow wrote:
> Say I want to backup a directory every 15 minutes from cron.  I wrap cron in
> a script that does locking to ensure only one instance of the script runs at
> any one time.
> At some stage during the day, I have a file that is 1Gb that gets added to
> the directory that I am backing up, and then a lot of smaller files.
> The next rsync job starts copying the big file, but it may take an hour to
> transfer the big file across the network and whilst that is happening, the
> other smaller files will have to wait.  I no longer have my directory
> synchronised every 15 minutes.

It is physically can not be done, because your network throughput does not allow to sync
everything during 15 minute interval. Even if you sync small files, the bigger ones still
not synchronized.

But you can try to play with --max-size and --min-size parameters and start two jobs in parallel
(one for big files, other for small). Independend locking for each job type will be required.

More information about the rsync mailing list