rsync script for snapshot backups

[Petros Angelatos]

On 19 June 2016 at 10:27, Simon Hobson <linux at thehobsons.co.uk> wrote:
> Dennis Steinkamp <dennis at lightandshadow.tv> wrote:
>
>> i tried to create a simple rsync script that should create daily backups from a ZFS storage and put them into a timestamp folder.
>> After creating the initial full backup, the following backups should only contain "new data" and the rest will be referenced via hardlinks (-link-dest)
>> ...
>> Well, it works but there is a huge flaw with his approach and i am not able to solve it on my own unfortunately.
>> As long as the backups are finishing properly, everything is fine but as soon as one backup job couldn`t be finished for some reason, (like it will be aborted accidently or a power cut occurs)
>> the whole backup chain is messed up and usually the script creates a new full backup which fills up my backup storage.
>
> Yes indeed, this is a typical flaw with many systems - you often need to throw away the partial backup.
> One option that comes to mind is this :
> Create the new backup in a directory called (for example) "new" or "in-progress". If, and only if, the backup completes, then rename this to a timestamp. If when you start a new backup, if the in-progress folder exists, then use that and it'll be freshened to the current source state.

I have an extremely similar script for my backups and that's exactly
what I do to deal with backups that are stopped mid-way, either by
power failures or by me. I rsync to a .tmp-$target directory, where
$target is what I'm backing up. I have separate backups for my rootfs
and /home. I also start the whole thing under ionice so that my
computer doesn't get slow from all this I/O. Lastly, before renaming
the .tmp-$target to the final directory I do a `sync -f` because rsync
doesn't seem to call fsync() when copying files and you can have a
failed backup if a power failure happens after the rename().

Here is my script:

#!/bin/bash

set -o errexit
set -o pipefail

target=$1

case "$target" in
    home)
        source=/home
        ;;
    root)
        source=/
        ;;
esac

PATHTOBACKUP=/root/backup

date=$(date --utc "+%Y-%m-%dT%H:%M:%S")

ionice --class 3 rsync \
    --archive \
    --verbose \
    --one-file-system \
    --sparse \
    --delete \
    --compress \
    --log-file=$PATHTOBACKUP/.tmp-$target.log \
    --link-dest=$PATHTOBACKUP/$target-current \
    $source $PATHTOBACKUP/.tmp-$target

sync -f $PATHTOBACKUP/.tmp-$target

mv $PATHTOBACKUP/.tmp-$target.log $PATHTOBACKUP/$target-$date.log
mv $PATHTOBACKUP/.tmp-$target $PATHTOBACKUP/$target-$date

ln --symbolic --force --no-dereference $target-$date
$PATHTOBACKUP/$target-current