OK so I re-ran rdiff with several different block sizes against my two image files described earlier to try and find the optimal size with my use case. Here is a summary:<br> 64 byte block - signature size: 3199208028, delta size: 543335307<br>
256 byte block - signature size: 799802016, delta size: 267816685 <br> 512 byte block - signature size: 399901020, delta size: 220323243<br>1024 byte block - signature size: 199950516, delta size: 272722422<br>2048 byte block - signature size: 99975264, delta size: 446384815<br>
4096 byte block - signature size: 49987644, delta size: 830846129<br><br>At the risk of being obvious, this test showed that the smaller the block size specified, the longer the process takes to compute the signatures and deltas. It also validated tests done in rsync that show 512 is the best block size in my use case of transferring a Windows XP system image. <br>
<br>Most importantly, it clearly demonstrates that the rsync library is effective but there is a bug in rsync batch mode (both --write-batch and --only-write-batch) when using very large files.<br><br>What is the best way to file a bug report on this issue? Is there any other information needed, is there anyone reading this thread who is still not convinced there is a problem with rsync?<br>
<br>Thanks<br>Matt Van Mater<br><br><div class="gmail_quote">On Tue, Mar 20, 2012 at 4:26 PM, Matt Van Mater <span dir="ltr"><<a href="mailto:matt.vanmater@gmail.com">matt.vanmater@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I ran one more test on a separate VM to check and see if rsync would allow me to specify block size for a smaller file while using batch mode... it works. To me that indicates that rsync has a problem processing very large batch files, especially when you specify a particular block size. More signs point to an rsync bug...<br>
<br>Here are the commands to reproduce the 'success':<br>root@server:/images# echo foo > file1<br>root@server:/images# echo foo > file2<br>root@server:/images# echo bar >> file2<br>root@server:/images# rsync --only-write-batch=batch-2to1-512 --block size=512 file2 file1<br>
root@server:/images# ./batch-2to1-512.sh<br>root@server:/images# md5sum file1<br>f47c75614087a8dd938ba4acff252494 file1<br>root@server:/images#
md5sum file2<br>
f47c75614087a8dd938ba4acff252494 file2<br><br>I then increased the RAM on the VM to 32GB and then 64 GB and tried the command shown above on the big 16 GB files and it still fails with the same error as i originally reported. So even though rsync could fit the entire source, destination and diff file in RAM it still failed.<br>
<br>Still waiting on the rdiff with tiny block size to complete :)<span class="HOEnZb"><font color="#888888"><br><br>Matt</font></span><div class="HOEnZb"><div class="h5"><br><br><div class="gmail_quote">On Tue, Mar 20, 2012 at 4:09 PM, Joachim Otahal (privat) <span dir="ltr"><<a href="mailto:Jou@gmx.net" target="_blank">Jou@gmx.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Matt Van Mater schrieb:
<div><blockquote type="cite">Let me restate my last email regarding rdiff:<br>
<br>
All of my image files are from the same Windows XP VM, created
using FOG/partimage. Image1 is the "baseline", Image2 is Image1 +
the WinSCP binary downloaded (not even installed).<br>
</blockquote>
<br></div>
Use your virtualisation to create the difference / snapshot, and
transfer what your virtualisation spits out in the diff disk image,
and the let it merge on the target.<br>
How well that works depends on the bitchiness of your virtualisation
*g*.<br>
<br>
Good luck!<span><font color="#888888"><br>
<br>
Joachim Otahal</font></span><div><br>
<br>
<blockquote type="cite">
<br>
I am not imaging an Ubuntu machine. I am using the Ubuntu machine
as a means of creating the batch file for rsync and/or rdiff. I
chose that platform since it is a common distribution used by many
and would be easy for others to reproduce my problem.<br>
<br>
I agree the 400 MB still looks big, but no the ONLY intentional
difference between image1 and image2 is the 2.9 MB WinSCP binary i
downloaded. My guess is the difference is 1) due partially to the
default block size rdiff uses (512b?) AND 2) the fact that the
Windows XP VM image source only had 256 MB RAM and that by default
Windows XP creates a pagefile of 1.5 x RAM size = 384 MB. That is
close enough to 400 MB for me.<br>
<br>
I am currently running rdiff with a smaller blocksize to test #1
above, hopefully that will force the delta to get smaller (at the
expense of longer computation time).<br>
<br>
Matt<br>
<br>
<div class="gmail_quote">On Tue, Mar 20, 2012 at 3:41 PM, Joachim
Otahal (privat) <span dir="ltr"><<a href="mailto:Jou@gmx.net" target="_blank">Jou@gmx.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Matt Van
Mater schrieb:
<div><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Alternate assessment - I ran a similar comparison against
the two image files using rdiff that comes with Ubuntu
10.04.4 LTS (shown up as librsync 0.9.7) and have a
significantly smaller delta file (closer to what i
expect).<br>
</blockquote>
<br>
</div>
Just plain luck. If ubuntu wrote the most new files close to
the last used blocks and only changes a few bytes (this time
literally) in the middle then the desync happens later. The
400 MB delta still looks big, or did you install something big
like libreoffice?<br>
<br>
regards,<br>
<br>
Joachim Otahal<br>
<br>
</blockquote>
</div>
<br>
</blockquote>
<br>
</div></div>
</blockquote></div><br>
</div></div></blockquote></div><br>