Prepared meta directory for one to many sync

Lukas Macura macura at opf.slu.cz
Wed May 6 07:20:01 GMT 2009


Hi to all,

sorry, but I am not subscribed to this list. I have a question, because
I did not find any answer for my problem. If you have so, please let me
know! 

We use rsync to synchronise "one-to-many" workstations. We use our
obnovang software (http://obnova.bp.opf.slu.cz/). You can read more info
on this page, but in general, we are doing backups and refresh of many
workstations (aprox. 200) which are in our PC labs. So we prepare one
workstation, rsync data from local filesystem to rsync server and next,
all stations will connect to rsync server and refresh their local
filesystems to be copy of server.

Everything works great, thank you for great rsync code :) but we have
some suggestions how to improve speed of our process. 

- We cannot enable compression because file is compressed just when it
is sended. If 100PCs are connected to rsync server and we will turn on
compression, server will die. 
- We would want to use compression because some links are slow and
compression would increase speed.
- Same to partial file transfer. We use --whole-file , because if server
would compute hashes each time when client connect, server will die..

I know our scenario is similar to some rsync download mirror and maybe
it is solved somewhere. But I did not find any solution yet.

Maybe it would be great if rsync have some "meta" directory, where it
would prepare all infos about stored files. Like hashes, precompressed
files, etc. This "meta" directory could be created or refreshed by some
options. After creating and pointing to that directory, entire mirroring
process could be much more faster.

Maybe:
$ rsync -r --prepare-meta /tmp/meta ./
$ rsync -r --meta-dir /tmp/meta/ ./ rsync://somewhere/

Please is it theoreticaly possible ? Or is there some problem to
precompress and prehash files? 

Thank you for any answer,
Lukas Macura





More information about the rsync mailing list