Question about rsync and BIG mirror
jp at saucer.midcoast.com
Mon Mar 6 12:12:50 GMT 2006
100gb of 4-40MB files sounds like my home PC full of digital photos I've
taken. It backs up to a linux PC right beside it with rsync. I don't
really call it that big a project for rsync. Big things for rsync are
millions of files. At 100mbps, it takes a few seconds to build the list.
I use the -W option to copy the whole file instead of mess with
incremental on a fast network. The linux PC running rsync only has
768MB, athlon 1700, and runs X, so it's nothing special.
It will take a while over the VPN to build the file list and start
transferring, but it shouldn't be hard on the computer resouces.
I used to back the whole system up over a 512kbps link to our
datacenter, but don't anymore. That was reliable but obviously slow.
On Fri, Mar 03, 2006 at 08:02:55AM +0100, johan.boye at latecoere.fr wrote:
> // I wonder if this message has been posted, so I sent it again //
> I'm quite a n00b on rsync stuff but I went to the website, read
> FAQ/how-to, Google and more, I setup my own rsync server and clients:
> everything works fine :-D
> I'm preparing a plan for a production mode in my company: we need to
> mirror around 100GB of data trough a special VPN internet line 2MB
> The first time, the data will be transferred by a media such as a HD.
> Next, each night, we will try to update clients from the master server.
> It should be around 500MB to 3GB, no so much in comparison of the
> original size of data.
> I discovered "rsync" use a lot of CPU and RAM to run "checksums" on
> file that have to be synchronised. I need an opinion about my situation:
> So: each night, from 0:00am to maximum 7:00am, the server will have to
> check the 100Go of files and see what files have been modified, then,
> upload them to the clients. Each file is around 4MB to 40MB in average.
> I would like to know your opinion about this situation:
> - Should I setup a strong dual CPU computer dedicated to calculate this
> whole stuff?
> - What about the memory I should install?
> - Is there any bandwidth used during the checksums computation? Mine is
> quite limited.
> - I know the client computer will have to check files too; Disk I/O
> will be the most used. I think this computer will have NFS mount from a
> "datacenter" computer with a GB LAN card, I wonder it will be enough...
> I'm quite scared of the amount of data to check before synchronise
> clients, and how long it will take. To finish shortly, what do YOU
> think? Any advices?
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Jason Philbrook | Midcoast Internet Solutions - Internet Access,
KB1IOJ | Hosting, and TCP-IP Networks for Midcoast Maine
http://f64.nu/ | http://www.midcoast.com/
More information about the rsync