Rsyncing really large files

[Lars Karlslund]

Hello everyone,

I have a question regarding syncing of large files. We have a setup where
we need to sync a 500GB file with changes in random blocks.

The changes are made in a manner, so moving blocks is not probable
(actually it will never happen).

As I understand it, the way rsync works is:
- remote server calculates all checksum blocks and transmits these to client
- local client calculates all checksum blocks
- local client compares local blocks to remote blocks, checking if a block
could have moved
- changes are synced

Is it possible to have rsync calculate one block, transmit checksum, let
local decide to transmit or not, and then move on to next block?

The way it works now it takes a lot of time to actually transfer anything,
and uses a LOAD of memory (around 500MB). We're using 64K blocks.

If we do the rsync with "whole-files" option it takes 17 hours to complete
the sync.

Without "whole files" (checksumming enabled) it takes 5 days! (We've got
lots of bandwidth at the moment btw.)

Thanks!


-- 


Best regards,


Lars Karlslund
Pharma Nord
IT department