rsync using huge traffic

[Linda Walsh]

Ben Oswald wrote:
> Hey,
> 
> I'm using rsync to backup my server but there is a problem because
> rsync is using very huge amounts traffic. But first to the setup. The
> server I backup has 4GB of data and I use the following command to
> backup this data.  /usr/bin/rsync -aze 'ssh -i /root/.ssh/backup.key -l
> backupuser' --rsync-path='sudo rsync' --delete --exclude-from=ex.list
> $SRC $TRG 
---------

1) Just on a 'whim', Have you tried telling it NOT to use it's
differential algorithm and, instead, to send 'whole' files'?
(use --whole-file).

It seems obvious that any attempts to reduce traffic by doing compares,
in your situation, are unlikely to cause any benefit.

2) Second -- I notice that rsync has a deficiency in it's
archive option in that it doesn't preserve sparse files.

So you might add "--sparse" as well,

3) I don't know what you have in SRC and TRG -- you don't list them.

SRC could be //cifs/server/data, and TRG could be //cifs/server/backup-partition,
where /cifs/server/data would fetch the data from the server from
the backup system -- i.e. the //cifs/server/backup-partition would
store the data into a partition on the server, (like backup-partition),
which is really a drive on the backup server.  (I would think such would highly
unlikely, but if you had such an arrangement, then the data would be
transferred just to read the files would be ~3X your actual data and
to store it would be another 3-4X...  I.e. it could be hideous).

4)  If you are  doing a complete copy -- why are you using rsync?  It's
about the SLOWEST possible util you can find for doing a complete
file-sys -> file-sys copy.

Alternatives -- there are strengths and weaknesses
on some of these -- it depends on your needs.
(assuming you don't need ACL's or extended attr's as you aren't listing
them in your example);  As you don't use those, would guess you don't
need devices/pipes and such but just files and basic perms.

cd /server/data/root && {
	tar cfS - --exclude[several-exclude-ops to chose from] |
	ssh backupuser at backupserver bash -c '
	if [[ -d /backups/serverall ]]; then mv /backups/serverall /backups/serverall- 
&& rm -fr //backups/serverall- >&/dev/null & fi
	mkdir /backups/serverall && {
	cd /backups/serverall; tar xf -'msh


Others would be to use 'cpio' or 'star'.  cpio can pick up devices, pipes and such,
star in the -exastar -dump mode picks up all acl's and extended attrs.

By far, you fastest option, I'd think, would be to setup your backup device
as a samba server,  (or your server as a samba server and let a cron job on the 
backup
device read your files -- then you can use 'cp'.

I haven't done timing tests recently, but nfs at one point was better /faster for
linux->linux transfers with samba better for linux->win->linux, but both have 
evolved.

I know samba can R/W at near max theoretical (125MB-write, 119MB read) on 1Gb 
ethers, but haven't come close to figuring out best rates for 10-20Gb ethers.

----
5) That said, if you are determined to use rsync, .. why so cryptic an invocation?

why not:
export RSYNC_RSH=/usr/bin/ssh (or leave it at rsh as a default).
then:

rsync -a --one-file-system --delete-during --wholefile \
          --sparse  /server-root/. backupuser at backups:/backupdir/server-root/.

The "-z" option is generally worthless on Gigabit or faster ethernets.

Hope this gives you some ideas...

Linda