Initialising a mirror

[Cyrille Bollu]

Hi, 

thank you for your answer.

But what do you mean by "slow rsync"? Do you mean limiting bandwith? I 
think this will not be an option with 100Gb of data to synchronise...

However, I like your idea with the LAN: I think I will bring my mirror to 
the remote site :-)

Just for the record, I didn't made the standard mistake, having things one 
directory level off from where they should be.

I was hoping doing a "scp -rp" would preserve all the necessary 
information needed by rsync but apparently not... Maybe I should use tar?

Regards,

Cyrille

<Tony at ServaCorp.com> a écrit sur 19/10/2005 11:00:07 :

> Standard mistake is to somehow get things one directory level off 
> from where they should be.
> 
> If you have the available disk space (both ends)
> Do a fast rsync from the primary to locally available fast disk 
> space (same computer or LAN)
> Do a slow rsync from that disk space to another set of available 
> disk space over slow/unreliable network.
> Assuming the slow rsync completed without problems, 
>     do a fast rsync from the second disk space to the legitimate 
target..
> 
> Source makes a big distinction between foo/ and foo
> They both transfer the same files.
> They differ in where they transfer the file to  (BY ONE LEVEL)
> Both make sense and are quite reasonable.
> If you do both, the effects are similar to untarring twice with one 
> level difference between them.
> 
> I know I've done it a number of times, and it is no fun over remote 
> and often bad internet connections.
> With large volumes, it is not apparent when you've done it to yourself.
> 
> 
> WARNING  -- these scripts work but are probably not the best nor the
> most aesthetic.
> ---just a wee bit too informative and I'm too lazy to sanitize so 
> I'm not posting to the list.---
>     ?? hosts allow/deny ignore with auth users present ?? 
> The major sucurity concerns are NOT permanently losing all my data 
> rather than eavesdroppers.
> The internet connection(s) are ocassionally flaky to non-existent, 
> and everything needs to assume
> that they will chose the most inopportune moment to do something I don't 
like.
> Essentially the stunt is that /home/rsync-foo is rsync'd with other 
> such over slow and bad connections.
> With bad internet I've got (not shown) multiple off-site backups AND
> the ability to bring them all upt-to-date
> if there is a connection (any, not necessarily the preferred -- some
> connect, some don't connect)
> 
> 
> 
> ---Timing depends on what is different.  Typically this one is about
> 10 to 15 minutes  (about 10G) total.
> If large MySql tables are rearranged, tends to take rather longer.
> real    192m41.906s
> 
> 
> real    12m18.512s
> sent 13750 bytes  received 7577551 bytes  10279.35 bytes/sec
> total size is 14013850399  speedup is 1846.04
> Tue Oct 18 07:28:00 CDT 2005
> rsync-sjs-dwg
> [root at wf1 /root]# cat rsync-sjs-dwg
> #!/bin/sh
> 
> rsync client
> # staged backup
> time rsync -a /home/rsync-sjs-dwg/* /home/rsync-sjs-dwg-1bk/
> 
> time rsync -avz --progress --partial --timeout=1750 --password-
> file=/etc/rsync.secrets/rsync-sjs-dwg \
>         rsync-sjs-dwg at sjs::rsync-sjs-dwg/* /home/rsync-sjs-dwg/
> date; echo rsync-sjs-dwg
> 
> ----rsync server   (actually is an intermediate, not the primary)
> This is an extra "junk" computer with a decent hard drive.
> This serves to get a fast 
> # etc/rsyncd.conf
> pid file = /etc/rsyncd.pid
> [rsync-sjs-dwg]
>         comment = backup of sjs dwg
>         path = /home/rsync-sjs-dwg
>         use chroot = true
>         read only = false
>         list = false
>         uid = root
>         gid = root
>         auth users = rsync-sjs-dwg
>         secrets file = /etc/rsyncd.secrets
>         strict modes = true
>         hosts allow = 10.0.0.1 pete-162 wf
>         hosts deny = *.*.*.*
> ---rsync "client"  (This is a PRIMARY)
> [root at sys001 root]# cat rsync-sjs-dwg
> #!/bin/bash
> # rsync-sjs-dwg  SJS_Drawings/ title/
> mkdir -p /tmp/rsync ; echo `hostname` > /tmp/rsync/OPENED
> rsync -a --password-file=/etc/rsync.secrets/sjs-dwg  --timeout=750 \
>         /tmp/rsync/OPENED       rsync-sjs-dwg at 10.0.0.170::rsync-sjs-dwg/
> for name in title SJS_Drawings ; do
> rsync -a --password-file=/etc/rsync.secrets/sjs-dwg --timeout=750 \
>         /home/dwg/$name         rsync-sjs-dwg at 10.0.0.170::rsync-sjs-dwg/
> done
> mkdir -p /tmp/rsync ; echo `hostname` > /tmp/rsync/CLOSED
> rsync -a --password-file=/etc/rsync.secrets/sjs-dwg  --timeout=750 \
>         /tmp/rsync/CLOSED       rsync-sjs-dwg at 10.0.0.170::rsync-sjs-dwg/
> echo 'rsync-sjs-dwg'
> date
> date -u
> 
> 
> 
> 
> 
> -----Original Message-----
> From: rsync-bounces+tony=servacorp.com at lists.samba.org [mailto:
> rsync-bounces+tony=servacorp.com at lists.samba.org]On Behalf Of Cyrille 
Bollu
> Sent: Wednesday, October 19, 2005 2:58 AM
> To: rsync at lists.samba.org
> Subject: Initialising a mirror

> 
> Dears, 
> 
> I would like to mirror data from 2 servers connected together via 
> VPN over ADSL lines (dwn 2Mbps/Upld 512Kbps). 
> 
> I'm sure rsync is one of the best tool to keep these data in sync 
> but how should I use it to initialise the mirror? 
> 
> I'm currently testing the solution with 10Gb of data to keep in 
> sync. But on my lines it would take more than 40 hours to initially 
> create the mirror! And in production the data to keep in sync will 
> reach 100Gb... :-( 
> 
> So, I went with a laptop on the remote site and "scp -rp" the data 
> on it. I then got back to my site and "scp -rp" it on the mirror to 
> initialise it. 
> 
> But apparently I forgot something because when rsync still sends all
> the files from the remote site to my mirror when I first run it. 
> 
> Does anybody has an idea about what I'm doing wrong? 
> 
> Does anybody has a solution to this problem? 
> 
> Regards, 
> 
> Cyrille
-------------- next part --------------
HTML attachment scrubbed and removed