rsync with a single big file (100M)

Paul Slootman paul at debian.org
Wed Jan 24 11:51:29 GMT 2007


On Tue 23 Jan 2007, dejie wrote:
> 
> Anybody knows how rsync transfer one single file if there is difference
> between the two files, will rsync re-transfer the whole file or only
> transfer the difference? In my experiment, i found rsync would retranster

Only the difference.

> the whole file again if there's a difference.

No, your experiment does not show that...

> What I did were:
> I manually created a huge.txt (file size 100813288 ), run rsync command to
> transfer this file from server 1 to server 2(server2 does not have this file
> yet). The transfer time is 10 sec:
> Start Time: Thu Jan 18 17:06:26 SGT 2007
> building file list ... done
> huge.txt
> wrote 538885 bytes  read 101 bytes  51332.00 bytes/sec
> total size is 100813288  speedup is 187.04
> End Time: Thu Jan 18 17:06:36 SGT 2007
> 
> Append some text into huge.txt (now file size is 100819275) in server1, run
> rsync command to transfer this file from server1 to server2 (server2 has
> this file already at the moment). The transfer time is 17 sec:
>  
> Start Time: Thu Jan 18 17:08:33 SGT 2007
> building file list ... done
> huge.txt
> wrote 255 bytes  read 70395 bytes  4037.14 bytes/sec
> total size is 100819275  speedup is 1427.02
> End Time: Thu Jan 18 17:08:50 SGT 2007
> 
> As you can see that the second time transfer is much longer than the first
> time transfer, although there's only difference at end of the file. So i
> feel rsync retransfer the whole file if there's difference.

No, this only shows that the process took longer the second time. That's
because the file at the destination has to be read, checksums calculated
etc. to find the differences (and of course also the matches), and then
to build the new destination file and move it in place. However,
if you look at the byte counts:

> wrote 538885 bytes  read 101 bytes  51332.00 bytes/sec

vs.

> wrote 255 bytes  read 70395 bytes  4037.14 bytes/sec

you see that the second transfer did in fact transfer much less bytes
than the first time.


Rsync optimizes network traffic, not absolute speed. If your network is
fast enough, use --whole-file which will transfer the whole file which
will be faster in that case than finding and transferring the
differences, and merging those differences into the destination file.


Paul Slootman


More information about the rsync mailing list