rsync / checksum small block / xfer small block

[alexus]

not quite what i need

let's take another example

let's say i have a mysql db, and only few rows gets changed on daily
basis, yet that data file itself is huge, so rsync checks for checksum
sees that it's different and xfer the whole file, i use remote site,
so xfer takes long time and plus we pay for our bw, so it cost us
money too, instead there is a technology out there called dedupe, it
will slice file for many blocks ( i think 12k each ) and it will
checksum each of blocks, and will only trasnfer the one that were
changed, this way, if very little get changed during the day, the xfer
of large file will dramaticly drop, as it will only xfer one small
block 12k instead of 1G for instance...

so i was wondering if rsync ever going support this kind of
technology, as today there is more and more of data and not enough
time to back it up...




On Mon, Dec 1, 2008 at 10:13 PM, Matt McCutchen <matt at mattmccutchen.net> wrote:
> On Mon, 2008-12-01 at 21:51 -0500, alexus wrote:
>> is there a way to somehow force rsync to check checksum in blocks vs
>> the whole file and xfer only part that has changed vs the whole file?
>> for instance i have a few gig file, and it usually keeps adding at the
>> end, maybe a line a day, yet rsync xfer the whole file over and it
>> takes forever...
>
> The delta-transfer algorithm should limit the network traffic between
> the sending and receiving rsyncs, so I'm assuming your complaint is
> about disk I/O.  By default, the receiving side reads the entire old
> destination file to determine what blocks have changed and then rewrites
> the entire destination file.  --inplace will reduce the write to the
> changed blocks only, provided that the delta-transfer algorithm is
> enabled.  If you know that your files change only by appending, you can
> tell rsync that with the --append option and it will just copy the
> appended data.
>
> Matt
>
>



-- 
http://alexus.org/