Questions and comments regarding --remove-sent-files (Was: New delete option?)

[Evan Harris]

I've looked back through my mailing list archives, and seen a few messages 
touching on the same things I wanted to mention, but I figured it might be 
better to recap, since most of them were sent more than a year ago.

I have recently started using the --remove-sent-files option, and have 
noticed a couple of warts.  I'm using it to transfer (move really) gigabyte 
and larger sized files over a fairly slow connection (--bwlimit=10) and with 
keeping of partial files (--partial) to minimize transfer time in the event 
of connection problems.

Because the individual files may take a day or more each to transfer, rsync 
interruptions are not uncommon, and I've had several instances where the 
first run of a transfer aborted in the middle of the non-first file. 
Although rsync had successfully sent one or more files before losing the 
connection or being aborted, it doesn't appear to delete the files until a 
successful end of the whole rsync.  A later restart of the rsync sees that 
some of the files already exist on the destination and need no update, and 
those files get left on the sending side when they shouldn't.

So, I agree with the parent message that either --remove-sent-files should 
delete the files immediately after they are successfully sent, or a new 
option should be added (--move maybe?) that does it that way.

I saw a followup mailing list message from Wayne that suggested adding the 
-I option to cause the desired behavior, and that looks like it would be a 
good workaround.  Maybe all that is needed is to make a new --move option be 
an alias for --remove-sent-files and --ignore-times.  Would this be a fairly 
simple enhancement?

The other issue I wanted to touch on was also mentioned on the mailing list, 
and was how to guard against the possibility that files on the sending side 
might have been modified during the transfer (which for me sometimes takes a 
day or more), and for rsync to realize this and avoid deleting the file and 
losing those changes.  I know this one is a more difficult problem, but I 
just wanted to see if there might be an easy solution.

Wayne, thanks for all your work!

Evan


On Fri, 22 Apr 2005, Wayne Davison wrote:

> [It appears I missed this message back in February -- ouch.]
>
> On Sat, Feb 19, 2005 at 08:53:32PM -0500, Andrew Gideon wrote:
>> FWIW: In the manner I can envision using this, it makes more sense to
>> delete the source as long as the destination file is valid, whether
>> that file moved during this execution or not.  This provides a "mv"
>> function that's safe against a failure.
>
> That is an interesting point.  The current option allows you to have
> identical files that didn't get transferred, and thus don't get removed,
> but that does mean that if the transfer gets interrupted it might do the
> wrong thing with a file.
>
> I'll contemplate what to do going forward since the --remove-sent-files
> option was already released.  Perhaps a --remove-source-files option
> should be added that works as you suggested.
>
> ..wayne..
> -- 
> To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html