OK, I think I found the source of my problem...<br><br>One bug I found is due to the very large size of the file I am transferring (16993256652 image2 --> 17062442700 image1) combined with specifying a small-ish block size (512). The other bug I am unsure of the cause.<br>
<ol><li>Example 1 (works, but results in HUGE batch file size)</li><ol><li>root@ubuntu10_04_4:~# rsync --only-write-batch=img1-to-img2-defaultblock image2 image1</li><li>root@ubuntu10_04_4:~#ls -la | awk '/defaultblock/ {print $5 $8}</li>
<ol><li>7315408780 img1-to-img2-defaultblock<br></li></ol></ol><li>Example 2 (works, and results in a reasonable but not optimal batch file size)</li><ol><li>root@ubuntu10_04_4:~# rsync --block-size=1024 --only-write-batch=img1-to-img2-1024block image2 image1</li>
<li>root@ubuntu10_04_4:~#ls -la | awk '/1024block/ {print $5 $8}</li><ol><li>287002289 img1-to-img2-1024block<br>
</li></ol></ol><li>Example 3 (does NOT work, no batch file created)</li><ol><li>root@ubuntu10_04_4:~# rsync --block-size=512 --only-write-batch=img1-to-img2-512block image2 image1</li><ol><li>ERROR: out of memory in receive_sums [sender]</li>
<li>rsync error: error allocating core memory buffers (code 22) at util.c(117) [sender=3.0.7]<br></li></ol></ol></ol>
I am NOT a skilled C programmer and the following is a bit of a guess, but I think I have traced the bug back to sender.c in the function starting at line 59: static struct sum_struct *receive_sums(int f)<br><br>Something in those 59 lines of code does not like having to track so many blocks/signatures. I think it is the use of the int data type, and perhaps using int32 might solve my problem?<br>
<br>So I believe I have identified two distinct, but related bugs:<br>1) The use of the default block size when used in conjunction with batch mode results in an inexplicably large batch file size<br>2) The use of smaller block sizes with large files in conjunction with batch mode results in a failure, increasing block sizes to a larger value results in a success<br>
<br>Thoughts?<br><br>Matt Van Mater<br><br><br><div class="gmail_quote">On Wed, Mar 21, 2012 at 10:22 AM, Matt Van Mater <span dir="ltr"><<a href="mailto:matt.vanmater@gmail.com">matt.vanmater@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">OK so I re-ran rdiff with several different block sizes against my two image files described earlier to try and find the optimal size with my use case. Here is a summary:<br>
64 byte block - signature size: <a href="tel:3199208028" value="+13199208028" target="_blank">3199208028</a>, delta size: 543335307<br>
256 byte block - signature size: 799802016, delta size: 267816685 <br> 512 byte block - signature size: 399901020, delta size: 220323243<br>1024 byte block - signature size: 199950516, delta size: 272722422<br>2048 byte block - signature size: 99975264, delta size: 446384815<br>
4096 byte block - signature size: 49987644, delta size: 830846129<br><br>At the risk of being obvious, this test showed that the smaller the block size specified, the longer the process takes to compute the signatures and deltas. It also validated tests done in rsync that show 512 is the best block size in my use case of transferring a Windows XP system image. <br>
<br>Most importantly, it clearly demonstrates that the rsync library is effective but there is a bug in rsync batch mode (both --write-batch and --only-write-batch) when using very large files.<br><br>What is the best way to file a bug report on this issue? Is there any other information needed, is there anyone reading this thread who is still not convinced there is a problem with rsync?<br>
<br>Thanks<span class="HOEnZb"><font color="#888888"><br>Matt Van Mater</font></span><div class="HOEnZb"><div class="h5"><br><br><div class="gmail_quote">On Tue, Mar 20, 2012 at 4:26 PM, Matt Van Mater <span dir="ltr"><<a href="mailto:matt.vanmater@gmail.com" target="_blank">matt.vanmater@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I ran one more test on a separate VM to check and see if rsync would allow me to specify block size for a smaller file while using batch mode... it works. To me that indicates that rsync has a problem processing very large batch files, especially when you specify a particular block size. More signs point to an rsync bug...<br>
<br>Here are the commands to reproduce the 'success':<br>root@server:/images# echo foo > file1<br>root@server:/images# echo foo > file2<br>root@server:/images# echo bar >> file2<br>root@server:/images# rsync --only-write-batch=batch-2to1-512 --block size=512 file2 file1<br>
root@server:/images# ./batch-2to1-512.sh<br>root@server:/images# md5sum file1<br>f47c75614087a8dd938ba4acff252494 file1<br>root@server:/images#
md5sum file2<br>
f47c75614087a8dd938ba4acff252494 file2<br><br>I then increased the RAM on the VM to 32GB and then 64 GB and tried the command shown above on the big 16 GB files and it still fails with the same error as i originally reported. So even though rsync could fit the entire source, destination and diff file in RAM it still failed.<br>
<br>Still waiting on the rdiff with tiny block size to complete :)<span><font color="#888888"><br><br>Matt</font></span><div><div><br><br><div class="gmail_quote">On Tue, Mar 20, 2012 at 4:09 PM, Joachim Otahal (privat) <span dir="ltr"><<a href="mailto:Jou@gmx.net" target="_blank">Jou@gmx.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Matt Van Mater schrieb:
<div><blockquote type="cite">Let me restate my last email regarding rdiff:<br>
<br>
All of my image files are from the same Windows XP VM, created
using FOG/partimage. Image1 is the "baseline", Image2 is Image1 +
the WinSCP binary downloaded (not even installed).<br>
</blockquote>
<br></div>
Use your virtualisation to create the difference / snapshot, and
transfer what your virtualisation spits out in the diff disk image,
and the let it merge on the target.<br>
How well that works depends on the bitchiness of your virtualisation
*g*.<br>
<br>
Good luck!<span><font color="#888888"><br>
<br>
Joachim Otahal</font></span><div><br>
<br>
<blockquote type="cite">
<br>
I am not imaging an Ubuntu machine. I am using the Ubuntu machine
as a means of creating the batch file for rsync and/or rdiff. I
chose that platform since it is a common distribution used by many
and would be easy for others to reproduce my problem.<br>
<br>
I agree the 400 MB still looks big, but no the ONLY intentional
difference between image1 and image2 is the 2.9 MB WinSCP binary i
downloaded. My guess is the difference is 1) due partially to the
default block size rdiff uses (512b?) AND 2) the fact that the
Windows XP VM image source only had 256 MB RAM and that by default
Windows XP creates a pagefile of 1.5 x RAM size = 384 MB. That is
close enough to 400 MB for me.<br>
<br>
I am currently running rdiff with a smaller blocksize to test #1
above, hopefully that will force the delta to get smaller (at the
expense of longer computation time).<br>
<br>
Matt<br>
<br>
<div class="gmail_quote">On Tue, Mar 20, 2012 at 3:41 PM, Joachim
Otahal (privat) <span dir="ltr"><<a href="mailto:Jou@gmx.net" target="_blank">Jou@gmx.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Matt Van
Mater schrieb:
<div><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Alternate assessment - I ran a similar comparison against
the two image files using rdiff that comes with Ubuntu
10.04.4 LTS (shown up as librsync 0.9.7) and have a
significantly smaller delta file (closer to what i
expect).<br>
</blockquote>
<br>
</div>
Just plain luck. If ubuntu wrote the most new files close to
the last used blocks and only changes a few bytes (this time
literally) in the middle then the desync happens later. The
400 MB delta still looks big, or did you install something big
like libreoffice?<br>
<br>
regards,<br>
<br>
Joachim Otahal<br>
<br>
</blockquote>
</div>
<br>
</blockquote>
<br>
</div></div>
</blockquote></div><br>
</div></div></blockquote></div><br>
</div></div></blockquote></div><br>