weird interaction between --delete-delay and --partial-dir when transfer is interrupted

[Shivkumar Venkatasubrahmanyam]

Michal Soltys wrote:
>>>
>
>>> Either way - as a workaround, you could drop delete-excluded, and
>>> then just finalize the thing with something like: find /bar/ -depth
>>> -name ".rsync-partial" -exec rm -rf "{}" \;
>>>
>
> I meant more like: -name ".rsync-partial" -prune -exec .....
>
> Shivkumar Venkatasubrahmanyam wrote:
>> Makes sense.  And this does seem like a corner case :)  I did some 
>> more testing along the lines you suggested ...
>>
>> mkdir src ; touch src/a
>> mkdir dst ; mkdir dst/.rp
>> rsync -a -vvvv --delete-excluded --delete-delay --delay-updates 
>> --partial-dir=.rp src/ dest/ >log 2>&1
>>
>> It seems this bug/corner case requires all the following conditions:
>> (1) [...]
>>
>
> I've straced it a bit - it looks like the flow is as follows:
>
> 1) with delete-excluded + delay-updates - .rp is deleted before 
> transfers (delete-excluded implies --delete-during, which in turn 
> behaves like per-dir --delete-before), then recreated when there's a 
> need to rename temporary file to that dir `rename(".a.7e7C6w", 
> ".rp/a"` then .rp is deleted as it's no longer needed, /unless/ it has 
> something left in it.
>
> 2) with delete-excluded + delete-delay - due to lack of delay-updates, 
> .rp is not needed by rsync at all, /unless/ the transfer is 
> interrupted (then .rp will be created, and parital file moved there). 
> Deletion happens only once, at the end, due to delete-delay.
>
> 3) with all the 3 above - situation similar to 1) happens, but at the 
> very end, and there's funny conflict - .rp will be deleted due to not 
> being needed anymore, but then delete-delay will take the action. 
> First trying to delete .rp's contents, but ... the dir is not there 
> anymore, thus ENOENT (due to opendir failing) and rsync's exit code 23.
>
This confirms my suspicions - there are two separate attempts to delete .rp/
>> I use all three of these options: --delete-excluded because I have 
>> added "--filter" options over time and I want the destination to 
>> reflect these newly excluded files,
>
> Any particular reason for that ? Sender/receiver rules can achieve the
> same, with much nicer flexibility. What delete-exclude does, is just 
> turn exclude into sender-exclude globally. You can do that explicitly, 
> e.g.
>
> H, something
> -,s something
>
Ah!  I can achieve what I need by replacing --delete-excluded with -f 'H 
foo' -f 'H bar' ... But if I add -f 'R .rp/' to clean up leftover 
partial-dirs that are NOT used during the transfer to temporarily hold 
updates (and therefore not auto-removed due to the implicit protect rule 
rsync adds for the partial-dir), then the error (23) recurs.  In other 
words, if we have ...

src/
src/a
dst/
dst/a
dst/.rp

... then without -f 'R .rp/', dst/.rp is not removed.  With -f 'R .rp/', 
it is removed but if dst/a is updated then we have the same issue (exit 
with code 23).  This is entirely consistent with your strace i.e. code 
23 whenever there are two separate attempts to remove .rp/.  I admit its 
unlikely that dst/.rp would be leftover AND remain unused during a 
transfer since its very existence implies that a previous transfer did 
not complete and the next one will most likely use it to "resume" at the 
point of interruption.  The only exception would be if the file whose 
transfer was interrupted, was deleted on the source between the two 
rsync runs.  So, apart from some rare coincidences, I think your -f 'H 
something' tip should do the trick!

Thanks for all your help :)
Shiv