Batch mode creates huge diffs, bug(s)?

ericbambach1 at discover.com ericbambach1 at discover.com
Tue Mar 20 12:07:34 MDT 2012


Matt,

        Its probably not a rsync bug. Its likely that after booting to 
create the second image a large number of updates has happened at many 
different parts in the filesystem. You may have added only a few MB of 
data but a lot of little things are going on in an active system like 
filesystem timestamp updates, registry updates, etc. It could also have to 
do with the internal structure of the image. If it stores metadata about 
each part of the system the metadata could be different between runs 
causing a large number of differences.

        A 7GB diff of a 16GB file tells me about half the blocks were 
modified between runs which isn't completely unbelievable in an active, 
booted system.

Eric Bambach | Discover
Senior Assoc. Programmer, Warehouse Infrastructure and Tools
2500 Lake Cook Road, Riverwoods IL 60015
P: 224.405.2896 ericbambach1 at discover.com




From:   Matt Van Mater <matt.vanmater at gmail.com>
To:     <rsync at lists.samba.org>
Date:   03/20/2012 12:55 PM
Subject:        Batch mode creates huge diffs, bug(s)?
Sent by:        <rsync-bounces at lists.samba.org>



So the short summary of my problem is, the batch file rsync creates is 
HUGE for a very small change.  The idea is to create workstation image 
with partimage, update it with some software and send the image update 
diff over the wire to a large number of destinations over a satellite 
link, but the batch file updates are several orders of magnitude too 
large.  I don't know exactly how partimage creates image files, so the 
bytes/blocks may be ordered differently between my two variants but should 
be identical, so rsync _should_ be able to handle that right?

Software used: Ubuntu 9.10, fogproject.org v.28, partimage ??, rsync 3.0.6
Hardware: Running as VM in ESXi 4.1 U2, 4 x vCPU and 16 GB RAM, 200 GB 
disk (150+ GB free)

My testing process:
1.      Use FOG .28 / partimage to capture an image of and already 
configured Windows XP workstation
2.      Log in to workstation as normal user, download WinSCP (2.9 MB 
file), shut down machine gracefully
3.      Use FOG .28/partimage /  to capture the same system again, to a 
new image file.
4.      FOG uses gzip to compress the partimage file, and we need to 
compare uncompressed images
1.      Commands:
1.      mv image1 image1.gz && mv image2 image2.gz && gunzip image1.gz && 
gunzip image2.gz
2.      Resultant files:
1.      image1 size in bytes: 17,062,442,700
2.      image2 size in bytes: 16,993,256,652
3.      Difference in raw size in bytes: 69,186,048 (somewhat larger than 
the 2.9 MB difference I expect due to downloading WinSCP, but not the end 
of the world)
5.      Create rsync diff package
1.      Command:
1.      rsync –only-write-batch=img1toimg2_diff image2 image1
2.      Resultant files:
1.      img1toimg2_diff size in bytes: 7,315,408,780
2.      img1toimg2_diff.sh in bytes: 58
3.      Difference is WAY bigger than raw file size. This HAS to be a bug!
I thought perhaps specifying the block size might help (it does 
significantly in non-batch mode) but I get a error and cannot proceed.  I 
have tried in both rsync v3.0.6 and v3.0.7 to specify the block size, but 
the result is the same:
1.      Command:
1.      rsync --block-size=512 –only-write-batch=img1toimg2_diff image2 
image1
2.      Error message:
1.      ERROR: Out of memory in receive_sums [sender]
2.      rsync error: error allocating core memory buffers (code 22) at 
util.c(117) [sender=3.0.7]

I looked at the changelog and haven't seen any updates to util.c since 
rsync v3.0.6 was released that might address this issue.  So i think that 
I might be seeing two bugs: 1) huge diff size 2) crashing non-gracefully 
when trying to use block size with batch mode.

Has anyone experienced this before, am I allowed to specify block size 
with batch mode?  Any words of wisdom?

Thanks,
Matt Van Mater-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: 
https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Please consider the environment before printing this email.


More information about the rsync mailing list