[Samba] Samba AD/DC crashed again, third time in as many months

[Andrew Bartlett]

On Wed, 2016-03-02 at 16:42 -0500, IT Admin wrote:
> I built this machine, and while it isn't the most robust box in the
> world
> it has been stable otherwise.  The RAID array is configured RAID1, I
> can't
> see how that could cause corruption issues and I haven't experienced
> any
> other data corruption issues apart from SAMBA collapsing

I know it is hard to swallow, but I really think this is hardware, or
the OS configuration under it, combined with unexpected shutdown or
some other corruption vector.

We have at this stage 10,000 or more domains running Samba4, and this
is only the second I've heard of with this kind of symptom.  The first
I blamed on a use of DRDB that I postulated was not preserving 'write
barriers' (that is, the thing that makes fsync() work) and a poweroff,
but I didn't really have any proof.

You do need to run a second DC, as well as run tools like memcheck on
this DC.  Make sure you regularly run the backup script, so you can
work out when the corruption happens, and verify your DB with dbcheck.

The second DC has the advantage that this kind of low-level corruption
doesn't easily spread across DRS replication (it would instead fail
replication). 

The error shown indicates that for some reason or other, it can't read
the schema.  This is very odd, as the schema doesn't change!  

We would love to get to the bottom of this. 

Unlike others, I don't think this has anything to do with packaging
(that would just make us not start at all), but a clean install on a
clean machine is my best advise, keeping the rest aside (and off) for
forensics if you have the patience. 

Finally, always keep the steps simple - otherwise we might start
confusing admin errors for hardware errors or vice verca.  The things
we all do in the panic are always the hardest to de-construct in the
cold light of day.

Thanks,

Andrew Bartlett

-- 
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT   
https://catalyst.net.nz/services/samba