Initialising a mirror

[Tony at ServaCorp.com]

"slow rsync"    limited by bandwidth (or whatever) compared to....
FAST (LAN), SLOW (Internet), FAST (LAN)  from "pointA" to "pointB"

100Gb of data    -- how much/many of what is changing?
How many files?  How big? (ranges)  where changed?
How many (major) directories?
What kind of window and how big?
What are the preferable failure modes?
    You do NOT have a choice of whether it fails.
    You DO have a choice of which way it fails.

Looks like -p of scp preserves times and modes.  ?? owner and group ??

Probably worth doing a fast look to see just what it is that you actually
have on both ends.
Intentions and reality quite often manage to do a disconnect.
You tend to get better answers if you state what you did (cut&paste)
as opposed to what you thought you did.
Generally when you have troubles they are not the same.

-----Original Message-----
From: Cyrille Bollu [mailto:Cyrille.Bollu at fedasil.be]
Sent: Wednesday, October 19, 2005 4:35 AM
To: Tony at ServaCorp.com
Cc: rsync at lists.samba.org
Subject: RE: Initialising a mirror



Hi,

thank you for your answer.

But what do you mean by "slow rsync"? Do you mean limiting bandwith? I think
this will not be an option with 100Gb of data to synchronise...

However, I like your idea with the LAN: I think I will bring my mirror to
the remote site :-)

Just for the record, I didn't made the standard mistake, having things one
directory level off from where they should be.

I was hoping doing a "scp -rp" would preserve all the necessary information
needed by rsync but apparently not... Maybe I should use tar?

Regards,

Cyrille

<Tony at ServaCorp.com> a écrit sur 19/10/2005 11:00:07 :

> Standard mistake is to somehow get things one directory level off
> from where they should be.
>
> If you have the available disk space (both ends)
> Do a fast rsync from the primary to locally available fast disk
> space (same computer or LAN)
> Do a slow rsync from that disk space to another set of available
> disk space over slow/unreliable network.
> Assuming the slow rsync completed without problems,
>     do a fast rsync from the second disk space to the legitimate target..
>
> Source makes a big distinction between foo/ and foo
> They both transfer the same files.
> They differ in where they transfer the file to  (BY ONE LEVEL)
> Both make sense and are quite reasonable.
> If you do both, the effects are similar to untarring twice with one
> level difference between them.
>
> I know I've done it a number of times, and it is no fun over remote
> and often bad internet connections.
> With large volumes, it is not apparent when you've done it to yourself.
>
>
> WARNING  -- these scripts work but are probably not the best nor the
> most aesthetic.
> ---just a wee bit too informative and I'm too lazy to sanitize so
> I'm not posting to the list.---
>     ?? hosts allow/deny ignore with auth users present ??
> The major sucurity concerns are NOT permanently losing all my data
> rather than eavesdroppers.
> The internet connection(s) are ocassionally flaky to non-existent,
> and everything needs to assume
> that they will chose the most inopportune moment to do something I don't
like.
> Essentially the stunt is that /home/rsync-foo is rsync'd with other
> such over slow and bad connections.
> With bad internet I've got (not shown) multiple off-site backups AND
> the ability to bring them all upt-to-date
> if there is a connection (any, not necessarily the preferred -- some
> connect, some don't connect)
>
>
>
> ---Timing depends on what is different.  Typically this one is about
> 10 to 15 minutes  (about 10G) total.
> If large MySql tables are rearranged, tends to take rather longer.
> real    192m41.906s
>
>
> real    12m18.512s
> sent 13750 bytes  received 7577551 bytes  10279.35 bytes/sec
> total size is 14013850399  speedup is 1846.04
> Tue Oct 18 07:28:00 CDT 2005
> rsync-sjs-dwg
> [root at wf1 /root]# cat rsync-sjs-dwg
> #!/bin/sh
>
> rsync client
> # staged backup
> time rsync -a /home/rsync-sjs-dwg/* /home/rsync-sjs-dwg-1bk/
>
> time rsync -avz --progress --partial --timeout=1750 --password-
> file=/etc/rsync.secrets/rsync-sjs-dwg \
>         rsync-sjs-dwg at sjs::rsync-sjs-dwg/* /home/rsync-sjs-dwg/
> date; echo rsync-sjs-dwg
>
> ----rsync server   (actually is an intermediate, not the primary)
> This is an extra "junk" computer with a decent hard drive.
> This serves to get a fast
> # etc/rsyncd.conf
> pid file = /etc/rsyncd.pid
> [rsync-sjs-dwg]
>         comment = backup of sjs dwg
>         path = /home/rsync-sjs-dwg
>         use chroot = true
>         read only = false
>         list = false
>         uid = root
>         gid = root
>         auth users = rsync-sjs-dwg
>         secrets file = /etc/rsyncd.secrets
>         strict modes = true
>         hosts allow = 10.0.0.1 pete-162 wf
>         hosts deny = *.*.*.*
> ---rsync "client"  (This is a PRIMARY)
> [root at sys001 root]# cat rsync-sjs-dwg
> #!/bin/bash
> # rsync-sjs-dwg  SJS_Drawings/ title/
> mkdir -p /tmp/rsync ; echo `hostname` > /tmp/rsync/OPENED
> rsync -a --password-file=/etc/rsync.secrets/sjs-dwg  --timeout=750 \
>         /tmp/rsync/OPENED       rsync-sjs-dwg at 10.0.0.170::rsync-sjs-dwg/
> for name in title SJS_Drawings ; do
> rsync -a --password-file=/etc/rsync.secrets/sjs-dwg --timeout=750 \
>         /home/dwg/$name         rsync-sjs-dwg at 10.0.0.170::rsync-sjs-dwg/
> done
> mkdir -p /tmp/rsync ; echo `hostname` > /tmp/rsync/CLOSED
> rsync -a --password-file=/etc/rsync.secrets/sjs-dwg  --timeout=750 \
>         /tmp/rsync/CLOSED       rsync-sjs-dwg at 10.0.0.170::rsync-sjs-dwg/
> echo 'rsync-sjs-dwg'
> date
> date -u
>
>
>
>
>
> -----Original Message-----
> From: rsync-bounces+tony=servacorp.com at lists.samba.org [mailto:
> rsync-bounces+tony=servacorp.com at lists.samba.org]On Behalf Of Cyrille
Bollu
> Sent: Wednesday, October 19, 2005 2:58 AM
> To: rsync at lists.samba.org
> Subject: Initialising a mirror

>
> Dears,
>
> I would like to mirror data from 2 servers connected together via
> VPN over ADSL lines (dwn 2Mbps/Upld 512Kbps).
>
> I'm sure rsync is one of the best tool to keep these data in sync
> but how should I use it to initialise the mirror?
>
> I'm currently testing the solution with 10Gb of data to keep in
> sync. But on my lines it would take more than 40 hours to initially
> create the mirror! And in production the data to keep in sync will
> reach 100Gb...

>
> So, I went with a laptop on the remote site and "scp -rp" the data
> on it. I then got back to my site and "scp -rp" it on the mirror to
> initialise it.
>
> But apparently I forgot something because when rsync still sends all
> the files from the remote site to my mirror when I first run it.
>
> Does anybody has an idea about what I'm doing wrong?
>
> Does anybody has a solution to this problem?
>
> Regards,
>
> Cyrille
-------------- next part --------------
HTML attachment scrubbed and removed