delta copies

[WebTent]

On Sun, 2007-09-23 at 12:09 -0400, Matt McCutchen wrote:
> On 9/23/07, Robert Fitzpatrick <lists at webtent.net> wrote:
> > On Sun, 2007-09-23 at 00:56 -0400, Matt McCutchen wrote:
> > > Either the delta transfer algorithm is not being used due to a
> > > misconfiguration, or the pgsql backups are changing in a perverse way
> > > that prevents it from matching any data.
> 
> > Doing some more testing this morning, maybe what you suggested about the
> > pgsql backup is what is happening.
> 
> The "perverse way" I mentioned would be along the lines of updating a
> set of timestamps that appear every few hundred bytes in the backup
> file, regardless of how many values in the database have actually
> changed.  This would prevent any of the blocks into which rsync splits
> the old destination file from matching the source file.

Well, I am getting matched data, but it just doesn't seem to be matching
very much considering the small change in file size. I tested one dump
after another rsyncing in between dumps and got very little matched
data :(

esmtp# ls -la maia.sql
-rw-r--r--  1 root  wheel  997960610 Sep 23 14:14 maia.sql
mx1# pg_dump -Fc -Upgsql maia > maia.sql
mx1# ls -la maia.sql 
-rw-r--r--  1 root  wheel  999709040 Sep 23 15:55 maia.sql
mx1# rsync -az --stats --progress data/maia.sql esmtp:/data/backup/mx1.webtent.net/db/data/
building file list ... 
1 file to consider
maia.sql
   999709040 100%  369.39kB/s    0:44:02 (xfer#1, to-check=0/1)

Number of files: 1
Number of files transferred: 1
Total file size: 999709040 bytes
Total transferred file size: 999709040 bytes
Literal data: 987800910 bytes
Matched data: 11908130 bytes
File list size: 35
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 987566347
Total bytes received: 221228

sent 987566347 bytes  received 221228 bytes  371000.03 bytes/sec
total size is 999709040  speedup is 1.01

Since this database is very active as a mail cache, I guess it is
changing more data than it seems.

-- 
Robert