rsync script for snapshot backups
petrosagg at gmail.com
Wed Jun 22 05:14:26 UTC 2016
On 19 June 2016 at 10:27, Simon Hobson <linux at thehobsons.co.uk> wrote:
> Dennis Steinkamp <dennis at lightandshadow.tv> wrote:
>> i tried to create a simple rsync script that should create daily backups from a ZFS storage and put them into a timestamp folder.
>> After creating the initial full backup, the following backups should only contain "new data" and the rest will be referenced via hardlinks (-link-dest)
>> Well, it works but there is a huge flaw with his approach and i am not able to solve it on my own unfortunately.
>> As long as the backups are finishing properly, everything is fine but as soon as one backup job couldn`t be finished for some reason, (like it will be aborted accidently or a power cut occurs)
>> the whole backup chain is messed up and usually the script creates a new full backup which fills up my backup storage.
> Yes indeed, this is a typical flaw with many systems - you often need to throw away the partial backup.
> One option that comes to mind is this :
> Create the new backup in a directory called (for example) "new" or "in-progress". If, and only if, the backup completes, then rename this to a timestamp. If when you start a new backup, if the in-progress folder exists, then use that and it'll be freshened to the current source state.
I have an extremely similar script for my backups and that's exactly
what I do to deal with backups that are stopped mid-way, either by
power failures or by me. I rsync to a .tmp-$target directory, where
$target is what I'm backing up. I have separate backups for my rootfs
and /home. I also start the whole thing under ionice so that my
computer doesn't get slow from all this I/O. Lastly, before renaming
the .tmp-$target to the final directory I do a `sync -f` because rsync
doesn't seem to call fsync() when copying files and you can have a
failed backup if a power failure happens after the rename().
Here is my script:
set -o errexit
set -o pipefail
case "$target" in
date=$(date --utc "+%Y-%m-%dT%H:%M:%S")
ionice --class 3 rsync \
sync -f $PATHTOBACKUP/.tmp-$target
mv $PATHTOBACKUP/.tmp-$target.log $PATHTOBACKUP/$target-$date.log
mv $PATHTOBACKUP/.tmp-$target $PATHTOBACKUP/$target-$date
ln --symbolic --force --no-dereference $target-$date
More information about the rsync