Prepared meta directory for one to many sync

[Daniel.Li]

On Wed, 2009-05-06 at 09:20 +0200, Lukas Macura wrote:
> Hi to all,
> 
> sorry, but I am not subscribed to this list. I have a question, because
> I did not find any answer for my problem. If you have so, please let me
> know! 
> 
> We use rsync to synchronise "one-to-many" workstations. We use our
> obnovang software (http://obnova.bp.opf.slu.cz/). You can read more info
> on this page, but in general, we are doing backups and refresh of many
> workstations (aprox. 200) which are in our PC labs. So we prepare one
> workstation, rsync data from local filesystem to rsync server and next,
> all stations will connect to rsync server and refresh their local
> filesystems to be copy of server.
> 
> Everything works great, thank you for great rsync code :) but we have
> some suggestions how to improve speed of our process. 
> 
> - We cannot enable compression because file is compressed just when it
> is sended. If 100PCs are connected to rsync server and we will turn on
> compression, server will die. 
> - We would want to use compression because some links are slow and
> compression would increase speed.
> - Same to partial file transfer. We use --whole-file , because if server
> would compute hashes each time when client connect, server will die..

Why not try to use multi-server.  for example, this is a central server.

central server  ---> backup to three mirror server,named with #1, #2, #3

#1 mirror server ---> 50 PC
#2 mirror server ---> 50 PC
#3 mirror server ---> 50 PC
#4 mirror server ---> 50 PC

It might reduce the work load of central server. What do u think?
> 
> I know our scenario is similar to some rsync download mirror and maybe
> it is solved somewhere. But I did not find any solution yet.
> 
> Maybe it would be great if rsync have some "meta" directory, where it
> would prepare all infos about stored files. Like hashes, precompressed
> files, etc. This "meta" directory could be created or refreshed by some
> options. After creating and pointing to that directory, entire mirroring
> process could be much more faster.
> 
> Maybe:
> $ rsync -r --prepare-meta /tmp/meta ./
> $ rsync -r --meta-dir /tmp/meta/ ./ rsync://somewhere/
> 
> Please is it theoreticaly possible ? Or is there some problem to
> precompress and prehash files? 
> 
> Thank you for any answer,
> Lukas Macura
> 
> 
> 
-- 
USI/SH RD Dept.
Universal Scientific Industrial (Shanghai) Co., Ltd
Fax: +86-21-58967931    Tel: +86-21-58966996  ext. 1400
42th building, 1387 Zhang Dong road, Pudong New Area, Shanghai, P.R.
China 201203