[clug] Story: Fijian Resort complex loses a single disk: business process stops for 1-2 days

[Andrew Janke]

On 25 July 2014 10:53, steve jenkin <sjenkin at canb.auug.org.au> wrote:
> If anyone has a good solution to this question, I'd love to hear.
> I know it has to be simple, because adding complexity will not address the root causes of failed monitoring.

I don't think there is an answer beyond "hire good people", if you
can't or don't trust yourself, you're probably better off trusting
Microsoft or Goog's clouds.

Witness a small business friend of mine who bought an external WD 4
disk RAID thingy to backup. They were backing up regularly, however
when it failed I just couldn't figure out why the internal linux SW
raid was so messed up. I eventually discovered after much questioning
that each month they would pull a drive and put it in a bank vault and
swap with last months as an offsite backup. Yes, while it was running.

In this case they'd pulled a second drive and put it in an external
case as they were holidaying and had work to do. Astoundingly this was
working, somehow the firmware in the external drive recognised it and
mounted it as a SMB share, it too was a WD NAS thing. They had come
back home, plugged it all back in "to get everything up to date" and
perhaps because it was running, one of the remaining two drives
failed. We'll all have our opinions of this but in the end they were
earnestly doing what they thought was the right thing.

These people are now very happy with a google apps setup, it's a small
cost (certainly less than the WD thing), and it has backup included.
Yes there is still a risk as there is no guarantee from Google, but
the existing system had risk too.

At work we take a different approach, all data is backed up by a
person. It's their job. We don't backup linux desktops or servers,
they are all installed, configured and managed via FAI + cfengine2. If
one fails, we put in a new disk and re-image, or just re-image a new
one. So, I tell a lie, we backup one server, the FAI + cfengine
server! I've recently been looking at Ubuntu MAAS but haven't made the
jump yet.



a

PS: I have the WD thing sitting under my desk haunting me as a
reminder to myself to always check what clients do in your absence.
(in a nice constructive way).