Question about rsync and BIG mirror

[jp]

100gb of 4-40MB files sounds like my home PC full of digital photos I've 
taken. It backs up to a linux PC right beside it with rsync. I don't 
really call it that big a project for rsync. Big things for rsync are 
millions of files. At 100mbps, it takes a few seconds to build the list. 
I use the -W option to copy the whole file instead of mess with 
incremental on a fast network. The linux PC running rsync only has 
768MB, athlon 1700, and runs X, so it's nothing special.

It will take a while over the VPN to build the file list and start 
transferring, but it shouldn't be hard on the computer resouces.
I used to back the whole system up over a 512kbps link to our 
datacenter, but don't anymore. That was reliable but obviously slow.

On Fri, Mar 03, 2006 at 08:02:55AM +0100, johan.boye at latecoere.fr wrote:
> // I wonder if this message has been posted, so I sent it again //
> 
> Hello,
> 
>   I'm quite a n00b on rsync stuff but I went to the website, read
> FAQ/how-to, Google and more, I setup my own rsync server and clients:
> everything works fine :-D
> 
>   I'm preparing a plan for a production mode in my company: we need to
> mirror around 100GB of data trough a special VPN internet line 2MB
> symmetric.
>   The first time, the data will be transferred by a media such as a HD.
> Next, each night, we will try to update clients from the master server.
> It should be around 500MB to 3GB, no so much in comparison of the
> original size of data. 
>   I discovered "rsync" use a lot of CPU and RAM to run "checksums" on
> file that have to be synchronised. I need an opinion about my situation:
> 
> 
>   So: each night, from 0:00am to maximum 7:00am, the server will have to
> check the 100Go of files and see what files have been modified, then,
> upload them to the clients. Each file is around 4MB to 40MB in average. 
> 
> I would like to know your opinion about this situation:  
>  - Should I setup a strong dual CPU computer dedicated to calculate this
> whole stuff? 
>  - What about the memory I should install? 
>  - Is there any bandwidth used during the checksums computation? Mine is
> quite limited.
>  - I know the client computer will have to check files too; Disk I/O
> will be the most used. I think this computer will have NFS mount from a
> "datacenter" computer with a GB LAN card, I wonder it will be enough...
> 
>   I'm quite scared of the amount of data to check before synchronise
> clients, and how long it will take. To finish shortly, what do YOU
> think? Any advices?
> 
> 
> Thanks,
> 
> Johan
> --
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
/*
Jason Philbrook   |   Midcoast Internet Solutions - Internet Access,
    KB1IOJ        |  Hosting, and TCP-IP Networks for Midcoast Maine
 http://f64.nu/   |             http://www.midcoast.com/
*/