Rsyncing really large files
lak at pharmanord.com
Thu Feb 24 13:01:59 GMT 2005
I have a question regarding syncing of large files. We have a setup where
we need to sync a 500GB file with changes in random blocks.
The changes are made in a manner, so moving blocks is not probable
(actually it will never happen).
As I understand it, the way rsync works is:
- remote server calculates all checksum blocks and transmits these to client
- local client calculates all checksum blocks
- local client compares local blocks to remote blocks, checking if a block
could have moved
- changes are synced
Is it possible to have rsync calculate one block, transmit checksum, let
local decide to transmit or not, and then move on to next block?
The way it works now it takes a lot of time to actually transfer anything,
and uses a LOAD of memory (around 500MB). We're using 64K blocks.
If we do the rsync with "whole-files" option it takes 17 hours to complete
Without "whole files" (checksumming enabled) it takes 5 days! (We've got
lots of bandwidth at the moment btw.)
More information about the rsync