Is rsync right for us?

Thu Mar 31 05:43:42 GMT 2005

-a archive
DOS:	Cheap and effective stunt for the benefit of an incremental archive
program.
	Write the files that have the bit turned on, then turn off the bit.
	bits: Archive, ReadOnly, Hidden, System  (in some order)
	There is nothing comparable in UNIX
RSYNC:   Make an archive copy
	(preserve the UNIX attributes: Owner, group, permissions, timestamps)
	as opposed to date it NOW and owned by YOU.  (or something similar)
same word. Nothing in common in the meaning.

Based on some messes of directories over bad internet connections,
(and plenty of stumbling around)
You can probably "get away with" transferring every 30 minutes or more
often.

Disk is cheap. Make sure you have SEPARATE BACKUP COPIES, at least 2
independent,
THAT ARE USEABLE. You will eventually run into something where the syncs
come at an
inoportune moment when same file is in the middle of being written and
rsynced
to the other system at the same time it is in the middle of being written
on the other system. Something in the wrong place at the wrong time.

Cheap shot. Follow the rsync immediately with an identical rsync.
(Effective way to get a copy of MyISAM tables from a running system)
This cuts down but does not eliminate the window of opportunity
for Mother Nature to inflict Murphy's Law on you.

To simplify coordination, I would run the SERVER on one machine
and the SCRIPTS on the other.
1. PUSH updates from HERE to THERE
2. PULL updates from THERE to HERE

If you like playing with risks,
balance the risk of an obsolete file on the local machine
versus the risk of being the victim of bad timing and destroying the file.
Balanced with the risk of not finding out you really need a backup copy soon
enough.
Rsync and cheap disk, lots of cheap disk is your friend.

If you do not like playing with risks, Amadeus R. is giving you very sound
advice.

This is why I like rsync. It may give you a sense of proportion or how hard
it can be pushed.
It has seemed stable in the sense that it tends to right itself when
something rocks the boat.
I MIGHT run into a problem with an rsync bug. I WILL run into problems with
internet.
Relevant data:
	12.6G+	50,000+ files  11 files changed or new
	Aprox 2.1M bytes changed  (2101460 bytes received)
	0m13.706s wall time  HOST SYSTEM ---> 100Mbps FDX ---> RSYNC SYSTEM
	1m12.449s wall time  RSYNC ---> 1BACK same IDE drive.
	14m52.839s wall time RSYNC ---> BAD trans-Pacific internet connection --->
local system.

[edited and sanitized]	(and no apologies for bad code herein)
[root at sys001 root]# time ./rsync-XXX-dwg ; date ; date -u
real    0m13.706s
Thu Mar 31 12:10:57 CST 2005	C as in China, not Central
Thu Mar 31 04:10:57 UTC 2005

[root at sys001 root]# cat rsync-XXX-dwg
#!/bin/bash
# rsync-XXX-dwg  XXX_Drawings/ title/
mkdir -p /tmp/rsync ; echo `hostname` > /tmp/rsync/OPENED
rsync -a --password-file=/etc/rsync.secrets/XXX-dwg  --timeout=750 \
        /tmp/rsync/OPENED       rsync-XXX-dwg at 10.0.0.xxx::rsync-XXX-dwg/

for name in title XXX_Drawings ; do
rsync -a --password-file=/etc/rsync.secrets/XXX-dwg --timeout=750 \
        /home/dwg/$name         rsync-XXX-dwg at 10.0.0.xxx::rsync-XXX-dwg/
done

mkdir -p /tmp/rsync ; echo `hostname` > /tmp/rsync/CLOSED
rsync -a --password-file=/etc/rsync.secrets/XXX-dwg  --timeout=750 \
        /tmp/rsync/CLOSED       rsync-XXX-dwg at 10.0.0.xxx::rsync-XXX-dwg/

[root at sys001 root]# du -s /home/dwg/XXX_Drawings/
12618920        /home/dwg/XXX_Drawings

[root at xx1 /root]# time ./rsync-XXX-dwg ; date ; date -u
real    1m12.449s		<-- IDE to same IDE (lots of seeks?)
receiving file list ...
50246 files to consider

CLOSED
           7 100%    6.84kB/s    0:00:00  (1, 0.0% of 50246)
OPENED
           7 100%    0.49kB/s    0:00:00  (2, 0.0% of 50246)
[snip]

sent 420 bytes  received 2101460 bytes  2355.05 bytes/sec
total size is 12642359798  speedup is 6014.79
real    14m52.839s	<-- This is over a BAD internet connection*
Wed Mar 30 22:32:16 CST 2005	C as in Central, not China
Thu Mar 31 04:32:16 UTC 2005

*BAD Internet. 2 tries to SSH directly, both timed out trying to connect.
 (SSH'd to a box at big boss's home (DSL) and then from there. Different
routes)
 This is acceptable transfer over a sporadically available connection!

-----Original Message-----
From: rsync-bounces+tony=servacorp.com at lists.samba.org
[mailto:rsync-bounces+tony=servacorp.com at lists.samba.org]On Behalf Of
Amadeus R.
Sent: Wednesday, March 30, 2005 7:24 PM
To: rsync at lists.samba.org
Subject: Re: Is rsync right for us?

> Yes,
> We would want to overwrite the older file
> on the IBM server, to match the updated version
> on the G4.

This is possible, but I haven't tried that.

> I guess I mis-understood the -a option....
> :)

Sort of=)

> Does rsync need to be installed on both servers to work?

I don't know what is the best way, when you want to send data both ways.

Try reading:

http://rsync.samba.org/ftp/rsync/preview/rsync.html
http://rsync.samba.org/ftp/rsync/preview/rsyncd.conf.html

and look under CONNECTING TO AN RSYNC SERVER.

There are also good examples at
http://samba.anu.edu.au/rsync/examples.html

You mentioned you wanted to back up every 30minutes. Be sure that all
data can in worst case be transferred in less than 15 minutes to avoid
an never ending queue.

I would advise to only back up once per day for many reasons, but most
importantly, if you by mistake delete a file, you have to notice it
within 30 minutes before the file is deleted on the back up aswell.

Doing back up while someone is editing a file is never good, so try to
only make the back up, when the chance of someone working on the files
is low.

Daily Cron would be perfect for this job.

--
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html