[clug] Story: Fijian Resort complex loses a single disk: business process stops for 1-2 days

[Scott Ferguson]

On 26/07/14 07:32, Alex Satrapa wrote:
> On 25 Jul 2014, at 23:31, Scott Ferguson
> <scott.ferguson.clug at gmail.com> wrote:
> 
>> It's also good practise to test backup medium themselves - to see
>> how reliable they are at "keeping" verified backups.
> 
> One simple option is to mount the backup & compare the source and
> backup.

How does that verify the backup medium? (as opposed to verifying *what*
is being backed up).

e.g.
SMART for drives (unreliable - one of the reasons for the practise of 2
independent verified backups using different methods to verified mediums
for both local and offsite archives)

QPxtool or dvdisaster to verify DVD mediums (CDs and DVDs have multiple
levels of redundancy "built-in" so without verifying you don't know how
much error protection you have between you and recovery failure/data
corruption). dvdisaster also adds additional ECC.


> This can get rather tedious if (for example) you have several
> TB of data on a home network where the backups are stored on a USB
> disk attached to a consumer-grade NAS. It also means your backups
> will be offline for the duration. In my case this takes about a day,
> with the bottleneck being the single USB2 disk.

Ouch. I don't know your user-case so that's not a critisism, I have been
in a similar situation in the past and found it too cumbersome so now
make a practise of spreading the data and services across a number of
machines (I find VirtualBox appliances are very useful for that purpose)
instead of continually running. I also found the breaking the backup
into multiple backups makes the process easier (failures and errors are
less of a problem when they affect less files). From a "home user"
perspective that means that my Movies, Music, TV, Documents, Pictures,
References, and Settings are separate backups.

> 
> Alex
> 
> 
> 

In my case full backups are done far less regularly than incremental
backups. I use at least 3 backups (2 local, one offsite on DVDs).
Remote servers are rsynced to local development servers*1 (usually
virtualbox appliances) and backups are made from the local development
servers. The data centre that hosts the servers also provides a backup
service - but for several reasons I prefer not to rely on them (good for
quick restores though!). Don't trust what you can't verify, don't get
locked to a provider.

rsync with the checksum option enabled means I can 'prove' the backup
is, um, a backup *when I'm backing up*. But bandwidth constraints (and
dynamic data) means rsync is not a useful way of determining whether a
*previous* backup is still reliable. That's where gpg is useful... the
problem is that gpg won't generate a detached signature for multiple
files (i.e. a directory), my work around (and I welcome suggested
improvements) is to generate sigs for files in directories (gpg -abo
$i.sig --passphrase-file passphrase.txt $i) then generate a dir.sig for
those cat-ed sigs (file sigs in /tmp). The dir.sigs are timestamped
(date +%H%M-%d%m%Y) and emailed to me. The backup archives can then be
verified offline (without trying to sync to the original which has
surely changed) using gpg.

NOTE: when backing up entire drives the process is somewhat simpler as
LUKS is the default (the ability to unencrypt the backup 'proves' its
veracity)

Plenty of room for improvement, but (when properly documented) my
current procedures keep me happy.



Kind regards



*1
another lesson learned hard - *never* edit 'live' sites. Edit the local
development servers (export a copy of the VirtualBox machine as a full
backup before making the changes) and import changes (selectively rsync
file system changes, dump local dev dv and import to live). Changes have
then been tested *before* going live, changes are instant, and if/when
something doesn't work out as planned - restoring from the data centre
daily backup is fast and simple (usually).


--
_real_ men just upload their important stuff on ftp, and let the rest of
the world mirror it. -- Linus Torvalds on backups