rsync 3.1.1: --ignore-missing-args / --delete-missing args problem

[Georgy Fedorov]

Hi Alex,

Thank you very much indeed for that link. I was actually considering 
using find -0 together with --files-from, and making find to list only 
files that were changed some time ago -- but first, this would probably 
break hardlinks, and second, despite most of the files in question are 
temporary files, a user can still delete something old enough right in 
the middle of an rsync.

However, looking at rsync problems that we have more closely, I now 
realize that most of them are related to time setting issues -- part of 
these similar to bug https://bugzilla.samba.org/show_bug.cgi?id=4977 and 
the like. But I think these problems can and shall be addressed )

I will write more on this when I will finish testing.

Kind regards, George


On 08/04/17 00:01, Axel Kittenberger wrote:
> With this two options on a very live system you may need to take into 
> account this bug as well I reported a while ago:
>
> https://bugzilla.samba.org/show_bug.cgi?id=12569 
> <https://protect-au.mimecast.com/s/5vYEBwtZLRbJcm?domain=bugzilla.samba.org>
>
> Due to this I'm currently not using  --ignore-missing-args / 
> --delete-missing but rather -exclude=* -include=- and a autocreated 
> hierachical filter rule list on stdin.
>
> PS: Yes to simply ignore code 24 would be correct. However, it creates 
> exit code 2 in these cases.
>
> On Fri, Apr 7, 2017 at 2:58 PM, Georgy Fedorov via rsync 
> <rsync at lists.samba.org <mailto:rsync at lists.samba.org>> wrote:
>
>     Dear All,
>
>     We sometimes have to replicate large "live" filesystems with many
>     ( sometimes millions, up to few hundred millions ) files on them.
>     ( Copying actively used files is of course a bad idea, but it
>     really helps to keep the delta small, so one final transfer can
>     later save the day. )
>
>     The problem, as one may guess, is that some files may disappear
>     during the process, so rsync finishes with an error code 23.
>     The good news is that since version 3.1.0 rsync has two options to
>     address this problem: --ignore-missing-args and
>     --delete-missing-args .
>
>     The bad news, however, is that even with any of these, when the
>     file disappears, rsync handles the file transfer properly, but
>     then tries to set times or attributes on these files nevertheless.
>     That fails with errno 22 ( EINVAL ) and still leads to exit code
>     23, which is a bit of an annoyance.
>
>     I am currently trying to come up with a patch to address this
>     issue, since I'd like to have exit code 0 for either of
>     --ignore-missing-args or --delete-missing-args, when the files
>     are, well, missing.
>
>     Probably it is not the right way to address this problem, but in
>     the same vein as --ignore-missing-args are implemented, the patch
>     can go as follows:
>     https://gist.github.com/anonymous/96ba8bf10f864a93fd9203f75c43ffe9
>     <https://gist.github.com/anonymous/96ba8bf10f864a93fd9203f75c43ffe9> .
>
>     Basically we need to do two changes:
>
>     (1) in options.c, make sure that the "missing_args" value is
>     transferred when set ; and
>     (2) in rsync.c, in the "missing_args" case, replace FERROR_XFER
>     with something else, since apparently any log message with logcode
>     FERROR_XFER sets the flag 'got_xfer_error' in log.c, and that
>     finally leads to exit code RERR_PARTIAL (23), which is what we are
>     trying to avoid.
>
>     I am currently testing this on fairly big datasets to see if
>     there's something missing, and will write more when I see how it
>     goes ( as I said, the data sets are fairly large ).
>
>     There shall be a better way to address that, but this is all I can
>     do with a very shallow acquaintance with rsync source code.
>     Nevertheless, it would be great if that could be fixed in the
>     trunk one way or another )
>
>     Kind regards, George
>
>     -- 
>     George Fedorov
>     Senior Systems Specialist
>     Melbourne School of Engineering
>     The University of Melbourne, Victoria 3010, Australia
>     phone: *****
>     email: *****
>     http://www.eng.unimelb.edu.au/
>
>
>     --
>     Please use reply-all for most replies to avoid omitting the
>     mailing list.
>     To unsubscribe or change options:
>     https://lists.samba.org/mailman/listinfo/rsync
>     <https://protect-au.mimecast.com/s/gn98BxSeXmW9fJ?domain=lists.samba.org>
>     Before posting, read:
>     http://www.catb.org/~esr/faqs/smart-questions.html
>     <https://protect-au.mimecast.com/s/oDLrBgcVgWZDt8?domain=catb.org>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.samba.org/pipermail/rsync/attachments/20170409/a2948a6a/attachment.html>