Rsync backups over the internet

[Wayne Davison]

On Sat, May 06, 2006 at 09:50:58PM -0400, Srinivas Kotapally wrote:
> I am use rsync to backup a large filesystem over the internet.  I do an
> incremental backup of the 2 file systems once a day to keep an offsite
> backup.  However, this backup takes about 60 minutes.

The first thing I'd recommend is to switch from using --delete to using
--delete-during, as this will require less disk I/O on the receiver.  If
that doesn't speed things up enough, you need to do some testing to
figure out what factor(s) are limiting your speed.  For instance, run
"top" or some other CPU-monitoring tool during the rsync run to see if
your CPU is being maxxed out (e.g. a slower CPU can become saturated
with the task of running rsync's file-difference algorithm at the same
time that ssh is encoding (and compressing?) the data being sent.  You
also want to figure out how near capacity your disk I/O.  If you don't
have a good disk-I/O-monitoring tool, you can at least get an estimation
of how much time is spent in the filesystem scan by doing a large "find"
operation and using that to estimate the amount of time that rsync must
use just getting the list of filenames on the sending side.  Finally,
use the stats that rsync outputs about how much data is transmitted to
see if bandwidth might be your limiting factor.  Once you determine what
is the bottleneck for the backup, you can try to figure out if there is
a way to speed it up.

..wayne..