Incremental Diffs?

[Alberto Accomazzi]

Kim,

it looks like you're correct in your assesment of the problem
(i.e. the whole file is copied).  You don't say what version of rsync
you're running, but there may be a bug in the current version introduced
recently with the addition of the --no-whole-file option.
See the following message for a description of the problem and a patch:

   http://lists.samba.org/pipermail/rsync/2002-March/006526.html


-- Alberto


In message <Pine.BSF.4.44.0203171953090.17210-100000 at jinx.unknown.nu>, Kim Scarborough writes:

> Okay, this response is a bit delayed, so I'm including a recap:
> 
> > > > > I'm using it to backup files from one computer to another, and it
> > > > > works exactly as I thought it would, except that it seems to be
> > > > > copying entire files over when they've change rather than the
> > > > > differences.
> > > >
> > > > What specifically leads you to that conclusion?
> > >
> > > I have it set to extra verbose, and I've been watching the files transfer
> > > over. When I append 2K to a 100MB text file and re-rsync, it's pretty
> > > obvious it's transferring 100MB, not 2K + whatever overhead the diff take
s
> > > up.
> >
> > I wouldn't be so sure.  Add the option --stats to the rsync command line
> > and see what it says.  AFAIK those numbers are correct.
> 
> Today I did a very specific test and I'm positive it's behaving that way.
> Here's a transcript, sorry for the length:
> 
> -----------------------------------------------------------------------------
-
> 
> JINX: /root# ls -l /home/sluggo/temp
> total 1904
> -rw-------  1 sluggo  wheel  1934600 Mar 17 19:49 asciifile
> 
> JINX: /root# rsync -vvaRbS --backup-dir=/backups/rsync/old --delete \
> --rsh=ssh --exclude="*.core" --exclude=/etc/namedb/secondary \
> --exclude=/var/run --numeric-ids --stats --progress --bwlimit=100 \
> /home/sluggo/temp remoteserver:/backups/rsync/current
> rsync: open connection using ssh remoteserver rsync --server -vvblogDtprRS
> --bwlimit=100 --delete --numeric-ids --backup-dir /backups/rsync/old . /backu
ps/rsync/current
> rsync: building file list...
> 
> rsync: expand file_list to 4000 bytes, did move
> rsync: 4 files to consider.
> expand file_list to 4000 bytes, did move
> deleting in home/sluggo/temp
> home is uptodate
> home/sluggo/temp/
> /home/sluggo/temp/asciifile
>       131072   6%   76.13kB/s    0:00:23
>       524288  27%   76.18kB/s    0:00:18
>      1212416  62%   76.19kB/s    0:00:09
>      1605632  82%   76.18kB/s    0:00:04
>      1934600 100%   76.01kB/s    0:00:00
> backup_dir is /backups/rsync/old
> total: matches=0  tag_hits=0  false_alarms=0 data=1934600
> 
> 
> Number of files: 4
> Number of files transferred: 1
> Total file size: 1934608 bytes
> Total transferred file size: 1934600 bytes
> Literal data: 1934600 bytes
> Matched data: 0 bytes
> File list size: 103
> Total bytes written: 1935042
> Total bytes read: 36
> 
> wrote 1935042 bytes  read 36 bytes  73021.81 bytes/sec
> total size is 1934608  speedup is 1.00
> 
> JINX: /root# echo "13 more chars" >> /home/sluggo/temp/asciifile
> 
> JINX: /root# rsync -vvaRbS --backup-dir=/backups/rsync/old --delete \
> --rsh=ssh --exclude="*.core" --exclude=/etc/namedb/secondary \
> --exclude=/var/run --numeric-ids --stats --progress --bwlimit=100 \
> /home/sluggo/temp remoteserver:/backups/rsync/current
> rsync: open connection using ssh remoteserver rsync --server
> -vvblogDtprRS --bwlimit=100 --delete --numeric-ids --backup-dir
> /backups/rsync/old . /backups/rsync/current
> rsync: building file list...
> 
> rsync: expand file_list to 4000 bytes, did move
> rsync: 4 files to consider.
> expand file_list to 4000 bytes, did move
> deleting in home/sluggo/temp
> home is uptodate
> /home/sluggo/temp/asciifile
>       131072   6%   76.19kB/s    0:00:23
>       425984  22%   76.19kB/s    0:00:19
>       851968  44%   76.18kB/s    0:00:14
>      1376256  71%   76.19kB/s    0:00:07
>      1934614 100%   76.02kB/s    0:00:00
> backup_dir is /backups/rsync/old
> keep_backup home/sluggo/temp/asciifile ->
> /backups/rsync/old/home/sluggo/temp/asciifile
> 
> total: matches=0  tag_hits=0  false_alarms=0 data=1934614
> 
> Number of files: 4
> Number of files transferred: 1
> Total file size: 1934622 bytes
> Total transferred file size: 1934614 bytes
> Literal data: 1934614 bytes
> Matched data: 0 bytes
> File list size: 103
> Total bytes written: 1935056
> Total bytes read: 36
> 
> wrote 1935056 bytes  read 36 bytes  70366.98 bytes/sec
> total size is 1934622  speedup is 1.00
> 
> --------------------------------------------------------------
> 
> Note the "Total bytes written". All I added was that one line at the end
> of the file. I also hit enter a couple times during the transfer so that
> snapshots of the timings would be logged; note that it took about the same
> amount of time both times. So unless I'm completely on drugs here, it
> seems to be transferring the entire file, not a diff.
> 
> -----------------------------------------------------------------------------
--
> Kim Scarborough                                      http://www.unknown.nu/ki
m/
> -----------------------------------------------------------------------------
--
> "I was so naive as a kid, I used to sneak behind the barn and do nothing."
>                                                            -Johnny Carson
> -----------------------------------------------------------------------------
--
> 
> 
> -- 
> To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsy
nc
> Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html



****************************************************************************
Alberto Accomazzi                          mailto:aaccomazzi at cfa.harvard.edu
NASA Astrophysics Data System                      http://adsabs.harvard.edu
Harvard-Smithsonian Center for Astrophysics        http://cfawww.harvard.edu
60 Garden Street, MS 83, Cambridge, MA 02138 USA   
****************************************************************************