--link-dest --inplace updates files without unlinking. What to do?

Askar Safin safinaskar at mail.ru
Thu Dec 25 19:49:17 MST 2014


Hi. This is bug report and simultaneously urgent asking for help. I am trying to write my rsync wrapper script, which will create minutely snapshots of my data using --link-dest. I want this script to be robust, it should work even if I do suspend/hibernate/reboot without notifying the script about this actions, it should work if I make hard-reset of the computer and if I disconnect network. Also this script should transfer large files with unstable network link. I. e. I want the following: I connected internet, it transferred some part of my large file, I disconnected inet, then I connected again, the script continued transferring. So, I wrote the following script (this script should be run locally, i. e. on my home computer with my data):

set +e

while :; do
        if rsync --archive --copy-unsafe-links --no-owner --no-group --one-file-system --compress --progress --inplace --append-verify --partial --delete-after --fuzzy --timeout=60 --link-dest=../latest-complete "$DIR/sync" "$HOST":backups/in-progress; then
                TIME="$(TZ=UTC date +%F-%H%M%S)"

                ssh "$HOST" "mv -T ~/backups/in-progress ~/backups/$TIME && ln -sfn $TIME ~/backups/latest-complete" || :
        fi

        sleep 60
done

It seems for me that this script address all this issues and it is OK, right?

As I said, I want the script to work with large files with unstable connection, so I decided to use "--inplace --append-verify" option. Am I right? Or just "--partial --append-verify" without "--inplace" will go? How exactly "--partial" without "--inplace" works? Where partial data is stored, in what files? How rsync knows it should continue transfer? Will --partial work in case of suddenly local computer hard reset?

Recently I noticed the following: I have in "$HOST":backups/2014-11-22-003120 too new copy of some my file. I am sure. Also, this file has a lot of hard links. So, I think the following happened: at 2014-11-22 actual right copy of this file was created at backup host. Then, some more snapshots was created with hardlinks to this file. Then, rsync started creating new snapshot in "in-progress", it created one more hardlink to that file and then failed for some reason (reboot, inet disconnection, etc). Then the file was changed, and rsync started again. rsync continued to update "in-progress" dir. It noticed that the file changed and updated it *without unlinking it*. So, all hardlinks, including very old ones, was updated. And this was wrong.

So, what to do? I. e. how to fix the script? Or maybe this is a bug in rsync? I. e. "--link-dest --inplace doesn't unlink"? Some quick fix will be good.

Maybe I should remove --inplace and leave --partial? But will the script still be such robust in this case?
==
Askar Safin
http://vk.com/safinaskar
Kazan, Russia


More information about the rsync mailing list