Hi all,<div><br></div><div>rsync is a fantastic tool. :-) I'm blown away with what I've seen so far.</div><div><br></div><div>I have a question about --partial-dir transfers. I've read through this thread:</div>
<div><a href="http://lists.samba.org/archive/rsync/2011-July/026575.html">http://lists.samba.org/archive/rsync/2011-July/026575.html</a></div><div>...but while similar, I don't think it's quite the same, and I didn't find my answer there.</div>
<div><br></div><div>The short(ish) version:</div><div><br></div><div>1. Am I correct in inferring that when rsync sees data for a file in the --partial-dir directory, it applies its delta transfer algorithm to the partial file?</div>
<div><br></div><div>2. And that this is _instead of_ applying it to the real target file? (Not a nifty three-way combination.)</div><div><br></div><div>If so, it would appear that this means a large amount of unnecessary data may end up being transferred in the second sync of a large file if you interrupt the first sync. Is there an option or some such to address this? If not, would it be feasible to add? (Details on how I see that working below, and I may be able to pitch in.)</div>
<div><br></div><div>The long version:</div><div><br></div><div>Sometimes I need to sync very large files (VM disk images) using ssh, during an eight-hour time window. With my connection to the target server, eight hours is unlikely to be enough, so I'll have to interrupt the sync and continue it in the next day's window. Sometimes, the VM disk image will be changed again in the meantime, but this isn't necessary to trigger the behavior I mentioned above. (It is a case I'll have to handle.)</div>
<div><br></div><div>I've run a few experiments with rsync in this area, and it looks like it causes a fair bit of unnecessary data transfer.</div><div><br></div><div>Here's how I caused that:</div><div><br></div>
<div>
1. I created a file with 100,000 lines of text with exactly the same length, and put it in both the source and destination.</div><div><br></div><div>2. In the source copy, I modified the first 20K lines. So roughly 20% of the file has been changed. I didn't change the *length* of the lines (in any of these experiments), because I'm trying to emulate a VM disk file which is conveniently organized into fixed-size blocks.</div>
<div><br></div><div>3. I started a sync:</div><div><br></div><div>rsync -avr --partial-dir=.rstmp src username@server:/dest/</div><div><br></div><div>...and cancelled it part-way through. This leaves a partial file in my .rstmp directory as expected. (In my case, just the first few hundred lines.)</div>
<div><br></div><div>4. I restarted the sync, allowing it to complete.</div><div><br></div><div>The second sync ended up transferring nearly the entire file, basically the whole 100K lines minus the few hundred from the first sync. The 80K of unchanged lines were transferred, whereas if I hadn't interrupted the first sync, they wouldn't have been.</div>
<div><br></div><div>I followed up with this experiment:</div><div><br></div><div>1. Starting with a synced file, I changed 20K lines in the *middle* of the file rather than at the beginning.</div><div><br></div><div>2. I started a sync and cancelled it part-way through, after about the same amount of time as the previous experiment. This leaves a partial file in my .rstmp directory as expected -- but it's a LOT bigger, rsync has quite intelligently copied the unchanged beginning of the file locally on the target machine, up until the first change, and then transferred the changed data after that -- which is when I interrupted it.</div>
<div><br></div><div>3. I started the sync again and let it continue, and it sent all of the rest of the file, the vast majority of which was already present in the original target file.</div><div><br></div><div>In subsequent experiments, I was able to determine that if I changed part of the file that had already been transferred into the partial file (say, changing line 1 between steps 2 and 3 above), rsync was very smart about that, just transferring the changed bit without re-transferring everything in-between. That's why it seems to me it uses the full delta-transfer algorithm on the partial -- or at least some version of it.</div>
<div><br></div><div>All of this seems to suggest that the partial file is created by copying the target file up to the first change and then applying changes -- but that if you interrupt it, because the partial file is shorter than the source file, all of the remaining source file is transferred.</div>
<div><br></div><div>Armed with that information, I tried to box clever: I thought "If I know I'm going to be doing one of these big files, maybe I could just copy the target to the .rstmp on the target machine in advance, so the delta-transfer applies to it." Unfortunately, though, cancelling the transfer early truncates the partial file. Drat. It wouldn't have been particularly elegant, but still would have been a workaround for now.</div>
<div><br></div><div>If I'm right about all of the above (which I wouldn't put money on), it seems like it would be possible to address this in a logically simple way. Logically simple doesn't equate to being simple in code, of course. :-) The idea being, basically, that when referring to blocks in the target partial file (whether for determining the checksum of the block or transferring the data), if the target partial file is missing the block entirely, use the equivalent block from the actual target file -- so for checksum purposes, that tells us whether it changed, and for data transfer purposes if it didn't change, we know we can copy it locally on the target server.</div>
<div><br></div><div>If there isn't already an option to address this, would it be feasible to do? I may be able to pitch in if so.</div><div><br></div><div>Thanks in advance,</div><div><font face="tahoma, sans-serif">--</font></div>
<div><font face="tahoma, sans-serif">T.J. Crowder</font><br>
</div>