[clug] How to make my server robust for booting

[Tony Lewis]

On 12/9/19 12:01 pm, George at Clug via linux wrote:
> Tony,
>
> I started to ask myself, is it possible to just have grub and /boot on two RAID1 drives, then have root (/) in the RAID 6 array?  I guess so.  Since grub and boot are very small, and usually read only, then using standard SSDs should not be too much of an issue?
I'd be guessing so too, but I am not going for RAID6 (see below).
> I would be curious of your final configuration and success (particularly if your game enough to do testing by manually degrading the system (i.e. remove a drive out of each RAID and see if you can still boot, and then replace the drive and see if you can still boot).

Sure, happy to share more, and take more advice.

/ is RAID1 (two or three partitions)

/boot is RAID1 (two or three partitions)

swap is four to six separate partitions

My main data 'volume' will be separate ext4 partitions, and will use 
AUFS to combine them into one big volume.

'Quasi'-RAID6 will come from SnapRAID, which will use two parity drives 
to be able to recover from the loss of any two of the data+parity volumes.

The SnapRAID solution gives the following advantages:

  * you can scale up in the future far more easily than regrowing a
    RAID6 array.  This applies to adding more disks and increasing the
    size of each volume
  * you can have mismatching volume sizes, so long as the parity volumes
    are the largest ones

But it has the following disadvantage: taking a snapshot is a manual or 
scheduled process, and doesn't provide instant coverage like software 
RAID does.  Any changes to the file system in between 'syncs' would be 
lost if the drive carrying that data died

The use case for SnapRAID is optimal for large files that don't change 
often, so backups, photos.  It also fits well with the mantra that 'RAID 
(including SnapRAID) is not backup'.

My intended backup solution is borgbackup + rclone + backblaze. I 
especially like the 'mount' function of borg and rclone so you can mount 
a particular backup archive as a FUSE filesystem and poke around for a 
file.  A second killer borg feature is the pruning (e.g. keep 7 daily 
backups, 4 weekly backups 6 monthly backups and perpetual yearly backups).

> All this takes so much time. I spent about a week once replacing each of the 5400 RPM drives for 7200 RPM drives in a six drive RAID, and then changing from RAID 5 to RAID 6.  To sync a drive took about 8 hours to resync. I was not always around when the resync completed to immediately start the next drive, though I did my best to time things.
I feel ya, and it's even more constraining when you consider what to do 
when you outgrow the array.  If you don't have more disk slots, you need 
to replace *all* the disks in a long process, and you don't get any 
benefit until you get that last one in place.  That was the main driver 
for me considering SnapRAID.
> I like stability and therefore like simplicity

I lean towards optimising, and therefore horribly over-complicate some 
things.  It's a flaw that keeps jumping up to bite me.

Thanks for your input.

Tony