<div dir="ltr">Not to mention the fact that ZFS requires considerable hardware resources (CPU & memory) to perform well. It also requires you to learn a whole new terminology to wrap your head around it.<div><br></div><div>It's certainly not a trivial swap to say the least...<div><br></div><div>Thanks,</div><div><br></div><div>-Clint</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 6, 2015 at 9:12 AM, Ken Chase <span dir="ltr"><<a href="mailto:rsync-list-m829@sizone.org" target="_blank">rsync-list-m829@sizone.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">This has been a consideration. But it pains me that a tiny change/addition<br>
to the rsync option set would save much time and space for other legit use<br>
cases.<br>
<br>
We know rsync very well, we dont know ZFS very well (licensing kept the<br>
tech out of our linux-centric operations). We've been using it but we're<br>
not experts yet.<br>
<br>
Thanks for the suggestion.<br>
<br>
/kc<br>
<div class="HOEnZb"><div class="h5"><br>
On Mon, Apr 06, 2015 at 12:07:05PM -0400, Kevin Korb said:<br>
>-----BEGIN PGP SIGNED MESSAGE-----<br>
>Hash: SHA1<br>
><br>
>Since you are in an environment with millions of files I highly<br>
>recommend that you move to ZFS storage and use ZFS's subvolume<br>
>snapshots instead of --link-dest. It is much more space efficient,<br>
>rsync run time efficient, and the old backups can be deleted in<br>
>seconds. Rsync doesn't have to understand anything about ZFS. You<br>
>just rsync to the same directory every time and have ZFS do a snapshot<br>
>on that directory between runs.<br>
><br>
>On 04/06/2015 01:51 AM, Ken Chase wrote:<br>
>> Feature request: allow --link-dest dir to be linked to even if file<br>
>> exists in target.<br>
>><br>
>> This statement from the man page is adhered to too strongly IMHO:<br>
>><br>
>> "This option works best when copying into an empty destination<br>
>> hierarchy, as rsync treats existing files as definitive (so it<br>
>> never looks in the link-dest dirs when a destination file already<br>
>> exists)".<br>
>><br>
>> I was suprised by this behaviour as generally the scheme is to be<br>
>> efficient/save space with rsync.<br>
>><br>
>> When the file is out of date but exists in the --l-d target, it<br>
>> would be great if it could be removed and linked. If an option was<br>
>> supplied to request this behaviour, I'd actually throw some money<br>
>> at making it happen. (And a further option to retain a copy if<br>
>> inode permissions/ownership would otherwise be changed.)<br>
>><br>
>> Reasoning:<br>
>><br>
>> I backup many servers with --link-dest that have filesystems of<br>
>> 10+M files on them. I do not delete old backups - which take 60min<br>
>> per tree or more just so rsync can recreate them all in an empty<br>
>> target dir when <1% of files change per day (takes 3-5 hrs per<br>
>> backup!).<br>
>><br>
>> Instead, I cycle them in with mv $olddate $today then rsync --del<br>
>> --link-dest over them - takes 30-60 min depending. (Yes, some<br>
>> malleability of permissions risk there, mostly interested in<br>
>> contents tho). Problem is, if a file exists AT ALL, even out of<br>
>> date, a new copy is put overtop of it per the above man page<br>
>> decree.<br>
>><br>
>> Thus much more disk space is used. Running this scheme with moving<br>
>> old backups to be written overtop of accumulates many copies of the<br>
>> exact same file over time. Running pax -rpl over the copies before<br>
>> rsyncing to them works (and saves much space!), but takes a very<br>
>> long time as it traverses and compares 2 large backup trees<br>
>> thrashing the same device (in the order of 3-5x the rsync's time,<br>
>> 3-5 hrs for pax - hardlink(1) is far worse, I suspect a some<br>
>> non-linear algorithm therein - it ran 3-5x slower than pax again).<br>
>><br>
>> I have detailed an example of this scenario at<br>
>><br>
>> <a href="http://unix.stackexchange.com/questions/193308/rsyncs-link-dest-option-does-not-link-identical-files-if-an-old-file-exists" target="_blank">http://unix.stackexchange.com/questions/193308/rsyncs-link-dest-option-does-not-link-identical-files-if-an-old-file-exists</a><br>
>><br>
>> which also indicates --delete-before and --whole-file do not help<br>
>> at all.<br>
>><br>
>> /kc<br>
>><br>
><br>
>- --<br>
>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~<br>
> Kevin Korb Phone: <a href="tel:%28407%29%20252-6853" value="+14072526853">(407) 252-6853</a><br>
> Systems Administrator Internet:<br>
> FutureQuest, Inc. Kevin@FutureQuest.net (work)<br>
> Orlando, Florida <a href="mailto:kmk@sanitarium.net">kmk@sanitarium.net</a> (personal)<br>
> Web page: <a href="http://www.sanitarium.net/" target="_blank">http://www.sanitarium.net/</a><br>
> PGP public key available on web site.<br>
>~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~<br>
>-----BEGIN PGP SIGNATURE-----<br>
>Version: GnuPG v2<br>
><br>
>iEYEARECAAYFAlUirykACgkQVKC1jlbQAQc83ACfa7lawkyPFyO9kDE/D8aztql0<br>
>AkAAoIQ970yTCHB1ypScQ8ILIQR6zphl<br>
>=ktEg<br>
>-----END PGP SIGNATURE-----<br>
>--<br>
>Please use reply-all for most replies to avoid omitting the mailing list.<br>
>To unsubscribe or change options: <a href="https://lists.samba.org/mailman/listinfo/rsync" target="_blank">https://lists.samba.org/mailman/listinfo/rsync</a><br>
>Before posting, read: <a href="http://www.catb.org/~esr/faqs/smart-questions.html" target="_blank">http://www.catb.org/~esr/faqs/smart-questions.html</a><br>
<br>
</div></div><span class="HOEnZb"><font color="#888888">--<br>
Ken Chase - ken att <a href="http://heavycomputing.ca" target="_blank">heavycomputing.ca</a> Toronto Canada<br>
</font></span><span class="im HOEnZb">Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W.<br>
</span><div class="HOEnZb"><div class="h5">--<br>
Please use reply-all for most replies to avoid omitting the mailing list.<br>
To unsubscribe or change options: <a href="https://lists.samba.org/mailman/listinfo/rsync" target="_blank">https://lists.samba.org/mailman/listinfo/rsync</a><br>
Before posting, read: <a href="http://www.catb.org/~esr/faqs/smart-questions.html" target="_blank">http://www.catb.org/~esr/faqs/smart-questions.html</a><br>
</div></div></blockquote></div><br></div>