<div dir="ltr">Wayne:<div><br></div><div>Thank you for responding.</div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 13, 2014 at 8:37 PM, Wayne Davison <span dir="ltr"><<a href="mailto:wayned@samba.org" target="_blank">wayned@samba.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class=""><div class="gmail_quote">On Tue, Mar 11, 2014 at 3:11 PM, Doug Robinson <span dir="ltr"><<a href="mailto:doug.robinson@wandisco.com" target="_blank">doug.robinson@wandisco.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">I was wondering what folks thought of a proposal to enhance rsync to be able to create and maintain a cache of {filePath, 64-bit mtime, checksum} beforehand on both source and target systems and then use that cache later on when asked to sync the two systems together? </blockquote>
</div><br></div>See patches (in order of recommendation): db.diff, checksum-updating.diff, checksum-xattr.diff.</div></div></blockquote><div><br></div><div>Will do.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_extra">I personally use db.diff in one situation at work combined with a sqlite DB on the source and destination machines. You just need to periodically weed out any old inode values (via rsyncdb --clean /dirs) if things start to bloat. In the future I'd like to see the db.diff code included by default as loadable libraries, which would allow someone to install plain rsync and only also install sqlite-using rsync and/or mysql-using modules if they want the extra functionality. There is also a plan to eventually have the db code map the inodes in the db to paths for things like rename optimizations.<br>
</div></div></blockquote><div><br></div><div>Nice.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"></div>
<div class="gmail_extra">That said, all these patches currently do is cache checksums. The db patch's default strict checking only uses a cached inode's info if the size+mtime+ctime all match what we knew about the file when it was cached (which makes it pretty safe).</div>
</div></blockquote><div><br></div><div>Seems fine unless xattrs are in play. I saw your comment in a prior posting abut ctime - and it makes a lot of sense. Thank you.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"> If you switch to a more lax algorithm (no ctime) you need to be extra sure the files don't get updated in some way as to leave the file matching the laxer inode info (e.g. only let rsync make changes to the files and/or make sure that modify timestamps always increase so that there is no chance of accidentally matching an older inode record).<br>
</div></div></blockquote><div><br></div><div>We're not dealing with xattr so I plan on using the size+mtime+ctime match.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"></div>
<div class="gmail_extra">If you're wondering how an mtime-using algorithm helps your use case, keep in mind that the mtimes don't need to match between hosts, just between each host's files and its db cache (and any non-matching or missing ones get (re)computed to the new checksum).<br>
</div></div></blockquote><div><br></div><div>That was the way that I understood it - glad to read I'm on the right path.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"></div>
<div class="gmail_extra">I'll also point out that if you want to use sqlite, I recommend you use the very latest db.diff (from the git patches repo) since it has a change that alleviates locking contention between the multiple rsync processes in a single copy (you can't really share the db between simultaneous rsync copies due to sqlite's poor multi-process locking -- use mysql for that).<br>
</div></div></blockquote><div><br></div><div>I'll have to consider my use case(s) to determine the right of sqlite vs. mysql. Thank you for the head's up.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"></div>
<div class="gmail_extra">The rsyncdb manpage has info on initializing the db, noting mounts, maintenance, etc.<br></div><div class="gmail_extra"><br></div><div class="gmail_extra">The other patches might also be useful to you, so feel free to check them out: <a href="https://git.samba.org/?p=rsync-patches.git" target="_blank">https://git.samba.org/?p=rsync-patches.git</a></div>
</div></blockquote><div><br></div><div>Excellent. Thank you.</div><div><br></div><div>Doug</div></div>-- <br><div dir="ltr"><div>Doug Robinson</div><div><div style="color:rgb(80,0,80)"><div dir="ltr" style="color:rgb(34,34,34)">
<font color="#666666"><br>WANdisco</font> <font color="#ff9900">//</font> <i><font color="#666666">Non-Stop Data</font></i></div></div><div><div dir="ltr"></div></div></div><div><div><a value="+447966783417" style="color:rgb(17,85,204)"><br>
</a></div><div><font color="#cccccc">t.</font><font color="#999999"> </font><a value="+447966783417" style="color:rgb(17,85,204)">925-396-1125</a></div><div><font color="#cccccc">e. </font><a href="mailto:doug.robinson@wandisco.com" style="font-size:small;color:rgb(17,85,204)" target="_blank">doug.robinson@wandisco.com</a><br>
</div></div></div>
</div></div>
<br>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:13.513513565063477px;background-color:rgb(255,255,255)"><p style="font-size:13.333333015441895px">Join us in New York and San Francisco for <a href="http://www.wandisco.com/subversion-git-live-2014" style="color:rgb(17,85,204)" target="_blank">Subversion & Git Live 2014</a></p><p style="font-size:13.333333015441895px">Listed on the London Stock Exchange: <a href="http://www.bloomberg.com/quote/WAND:LN" style="color:rgb(17,85,204)" target="_blank">WAND</a></p><p style="font-size:13.333333015441895px"></p><p style="font-size:13.333333015441895px"><font color="#999999" size="1">THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, ("WANdisco") does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed.</font></p></div>