rsync backup - order by size

[Ty Miller]

Hey guys,

Just thought I'd let you know that this worked perfectly. 

If the rsync guys would like a copy of my backup script for their examples
page, or anything like that, then I am happy to sanitize it and pass it on. 

It performs the following functions;

The backup script implements a 31 day rotating incremental backup. The
incrementals will go into subdirectories named after the day of the month
(1,2,3,...,31) and the current full backup goes into a directory called "0".
(This was named "0" because the Windows version of rsync/cygwin has
limitations on the filename length.)

A log file is kept, which is parsed at the end of the backup to determine
whether any errors have occurred. If so, then an email is sent to the
administrator containing the backup errors.

The script uses the --partial option so that if a transfer is interrupted
then partially transferred files will be kept so they can be continued on
the next transfer. I also added a check for a "connection reset" error so
that the backup is automatically kicked off immediately if the connection
drops out and alerts the administrator.

Large files were causing smaller files to not be backed up daily cause the
large files were taking longer than a day to complete. It now performs two
passes of backups for small and large files. The small files get backed up,
and then the large files start getting transferred. The next time the backup
gets kicked off it kills the large file backup if it is still running,
alerts the administrator, and restarts the process again so that the small
files get backed up, then the large files backup starts off where it left
off.

The large files were also causing issues with network performance, so a
bandwidth limit has been implemented for both small and large file
transfers.

Thanks again for your help,
Ty


-----Original Message-----
From: hashproduct at gmail.com [mailto:hashproduct at gmail.com] On Behalf Of Matt
McCutchen
Sent: Thursday, 29 March 2007 12:18 PM
To: Ty Miller
Cc: rsync at lists.samba.org
Subject: Re: rsync backup - order by size

On 3/28/07, Ty Miller <tyronmiller at gmail.com> wrote:
> If I changed this to include the --min-size and --max-size so that there
are
> the following two rsync passes;
>
> - Empty the daily incremental backup dir
> - Backup small files to the "Current" dir, moving modified or deleted
files
> to the daily incremental dir
> - Backup large files to the "Current" dir, moving modified or deleted
files
> to the daily incremental dir
>
> How will this affect the reality of what happens? For example;
>
> - Will the first pass cause the Current dir to only contain small files,
> causing it to move the large files to the daily incremental dir? [...]

It's good that you bring this up, but I don't think it will be a
problem.  The --min-size and --max-size options tell rsync to skip
transferring regular files whose sizes are out of range, but those
files are still present in the file list.  Thus, the receiver will
know not to delete any of the files from the "Current" dir that exist
on the sender, even files that belong to the other pass.  The two
passes should give you the same result as the one pass currently does.

Matt