script to rysnc to spare hard drive

[Martin Schwenke]

>>>>> "Gary" == Gary Peltola <clist at www-hosting.net> writes:

    Gary> Bascially what i am looking at doing is mirroring the first
    Gary> drive to the 2nd drive, and once that intial one is moved
    Gary> over, have a cron run nightly (or when specified) that will
    Gary> update any modified files from the cron run. I am looking to
    Gary> have this as a perl script (or whatever) so i can easily
    Gary> port it to all my servers across my network. So that if hard
    Gary> drive 1 fails, i will just need to pop in hard drive two and
    Gary> be up and running within minutes rather then hours (my last
    Gary> outage was pretty major on my /boot partion and if had rysnc
    Gary> setup in the manor i am speaking of, downtime would have
    Gary> went from 8 hours to less then 30 minutes....)

    Gary> Does anyone have such examples of what i can do to
    Gary> accomplish this? 

Below is a couple of scripts that do this...  but wait there's still
more.  :-)

The first script does this:

* Backup filesystems on /dev/hda to /dev/hdb, which gets mounted under
  /backup.

* /dev/hdb is bigger than /dev/hda, so (backwards) rsync incrementals
  are done to /backup/incremental, which uses the left over space on
  /dev/hdb. 

* Filesystems on /dev/hdb are backed up to tape using "dump".  This is
  cool, because the system can be up and running (and busy) while this
  is happening, since the filesystems on /dev/hdb aren't active.

The 2nd script:

* Cleans up the incrementals, so there's always a certain percentage
  of free space.  When it needs space it removes one or more whole
  days worth of incrementals.

  Note that in this script there is *some* sanity checking to try and
  ensure that sensible stuff is removed.  However, you should always
  *BEWARE* of scripts that run as root and automatically recursively
  remove directories.

  Just to be sure, I've changed the line that is supposed to remove
  stuff so that it looks like:

	    echo "Not currently executing this command: rm -rf" "${this}"

  so it now has to be modified so that the "rm -rf ..." is actually
  activated...  :-)

Remember, no warranty and all that...

peace & happiness,
martin

--------8<---------8<-------- CUT HERE --------8<---------8<--------
#!/bin/sh

# Uncomment these for debugging.
#set -x
DEBUG=echo

# Backup disk /dev/hda to /dev/hdb and then backup /dev/hdb to tape:

# 1. Using rsync, backup each mounted filesystem /X to
#    /backup/current/X, which is mounted especially for the task.  Old
#    version of files are stored under
#    /backup/incremental/DATETIME/X/...

# 2. Each mounted filesystem /X has its corresponding
#    /backup/current/X backed up using dump.  The /incremental
#    filesystem is also dumped at the end.

# Note that the incrementals aren't currently cleaned up.  Once we get
# an idea of the size of incrementals, we can started removing any
# directories older than (say) 6 months.

incr=/backup/incremental
yip="${incr}/`date +'%Y/%m/%d/%H%M%S'`"
tape=/dev/ntape

filesystems=`mount -t ext2 | awk '{print $3}'`

do_rsync () {
    $DEBUG /usr/bin/rsync -axH --delete \
	--backup --backup-dir="${yip}$1" \
	"$1/" "/backup/current$1"
}

do_mt () {
    ${DEBUG} /bin/mt -f "${tape}" "$1"
}

do_label () {
    tmp="/tmp/bkup$$"
    date > "${tmp}"
    n=1
    for i in $* ; do
	echo "$n $i" >> "${tmp}"
	n=`expr $n + 1`
    done
    $DEBUG /bin/dd if="${tmp}" of="${tape}" bs=512 conv=sync
    #$DEBUG /bin/tar cvf "${tape}" "${tmp}"
    do_mt eof
    /bin/rm -f "${tmp}"
}

do_dump () {
    ${DEBUG} /sbin/dump -0 -b 32 -a -f "${tape}" "$1"
}

######################################################################

echo "Check backup partitions starting: `date`."

fscks="${incr}"
disks=""
tapes=""

for i in ${filesystems} ; do
    case "$i" in
	/var/spool/squid)
	    :
	    ;;
	/)
	    fscks="${fscks} /backup/current"
	    disks="${disks} /"
	    tapes="${tapes} /backup/current"
	    ;;
	*)
	    fscks="${fscks} /backup/current${i}"
	    disks="${disks} ${i}"
	    tapes="${tapes} /backup/current${i}"
    esac
done

for i in ${fscks} ; do
    ${DEBUG} /sbin/fsck -a $i
    if [ $? -gt 1 ] ; then
	echo "fsck of backup filesystem $i failed.  Quitting!"
    fi
done

echo "Check backup partitions finished: `date`."

######################################################################

echo "Disk backup starting: `date`."

${DEBUG} mount ${incr}

for i in ${disks} ; do
    b="/backup/current${i}"
    if ${DEBUG} mount "${b}" ; then
	echo " ${i}"
	do_rsync "${i}"
	${DEBUG} umount "${b}"
    else
	echo "*** Failed to mount backup volume \"${b}\""
    fi
done

${DEBUG} umount ${incr}

echo "Disk backup finished: `date`."

######################################################################

echo "Tape backup starting: `date`."

do_mt rewind
# Eek! Tape drive doesn't do hardware compression.
#do_mt datcompression

do_label ${tapes}

for b in ${tapes} ; do
    if ${DEBUG} mount "${b}" ; then
	echo " ${b}"
	do_dump "${b}"
	${DEBUG} umount "${b}"
    else
	echo "*** Failed to mount backup volume \"${b}\""
    fi
done

do_mt offline

echo "Tape backup finished: `date`."

# Last line.
--------8<---------8<-------- CUT HERE --------8<---------8<--------
#!/bin/sh

# Cleanup the incremental backups on /backup/incremental to create
# free space.

base=/backup/incremental
threshold=90

mount $base

dirs=`echo /backup/incremental/*/*/*`

percent_used=`df /backup/incremental/ | tail -1 | sed -e 's/^.*[^0-9]\([0-9][0-9]*\).*$/\1/'`

while [ \( "${percent_used}" -gt ${threshold} \) -a -n "$dirs" ] ; do

    this=`echo $dirs | sed -e 's/ .*$//'`
    dirs=`echo $dirs | sed -e 's/^[^ ]* *//'`

    case "$this" in
	${base}/*/*/*)
	    echo "Removing old backup directory ${this}."
	    echo "Not currently executing this command: rm -rf "${this}"
	    ;;
	*)
	    echo "ERROR!!!  Bad name for backup directory: ${this}."
	    dirs=""
    esac

    percent_used=`df /backup/incremental/ | tail -1 | sed -e 's/^.*[^0-9]\([0-9][0-9]*\).*$/\1/'`
done

if [ "${percent_used}" -gt ${threshold} ] ; then
    echo "WARNING!!!"
    echo "${base} has ${percent_used}% used, which is greater than limit of ${threshold}%"
    echo "but there are no backups to remove!!!"
fi

umount $base

--------8<---------8<-------- CUT HERE --------8<---------8<--------