Batch mode creates huge diffs, bug(s)?

[Matt Van Mater]

So the short summary of my problem is, the batch file rsync creates is HUGE
for a very small change.  The idea is to create workstation image with
partimage, update it with some software and send the image update diff over
the wire to a large number of destinations over a satellite link, but the
batch file updates are several orders of magnitude too large.  I don't know
exactly how partimage creates image files, so the bytes/blocks may be
ordered differently between my two variants but should be identical, so
rsync _should_ be able to handle that right?

Software used: Ubuntu 9.10, fogproject.org v.28, partimage ??, rsync 3.0.6
Hardware: Running as VM in ESXi 4.1 U2, 4 x vCPU and 16 GB RAM, 200 GB disk
(150+ GB free)

My testing process:

   1. Use FOG .28 / partimage to capture an image of and already configured
   Windows XP workstation
   2. Log in to workstation as normal user, download WinSCP (2.9 MB file),
   shut down machine gracefully
   3. Use FOG .28/partimage /  to capture the same system again, to a new
   image file.
   4. FOG uses gzip to compress the partimage file, and we need to compare
   uncompressed images
      1. Commands:
         1. mv image1 image1.gz && mv image2 image2.gz && gunzip image1.gz
         && gunzip image2.gz
      2. Resultant files:
         1. image1 size in bytes: 17,062,442,700
         2. image2 size in bytes: 16,993,256,652
         3. Difference in raw size in bytes: 69,186,048 (somewhat larger
         than the 2.9 MB difference I expect due to downloading
WinSCP, but not the
         end of the world)
         5. Create rsync diff package
      1. Command:
         1. rsync –only-write-batch=img1toimg2_diff image2 image1
      2. Resultant files:
         1. img1toimg2_diff size in bytes: 7,315,408,780
         2. img1toimg2_diff.sh in bytes: 58
         3. Difference is WAY bigger than raw file size. This HAS to be a
         bug!

I thought perhaps specifying the block size might help (it does
significantly in non-batch mode) but I get a error and cannot proceed.  I
have tried in both rsync v3.0.6 and v3.0.7 to specify the block size, but
the result is the same:

   1. Command:
      1. rsync --block-size=512 –only-write-batch=img1toimg2_diff image2
      image1
   2. Error message:
      1. ERROR: Out of memory in receive_sums [sender]
      2. rsync error: error allocating core memory buffers (code 22) at
      util.c(117) [sender=3.0.7]


I looked at the changelog and haven't seen any updates to util.c since
rsync v3.0.6 was released that might address this issue.  So i think that I
might be seeing two bugs: 1) huge diff size 2) crashing non-gracefully when
trying to use block size with batch mode.

Has anyone experienced this before, am I allowed to specify block size with
batch mode?  Any words of wisdom?

Thanks,
Matt Van Mater
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20120320/fc5adee4/attachment.html>