[Samba] Samba AD/DC crashed again, third time in as many months

[IT Admin]

Well, this puts me in a catch-22 situation... I can see the benefit of
spinning up VMs as ADCs, unfortunately this machine is already leveraged to
the limit and there aren't any resources available to support a single
additional VM, let alone two of them.  And I'm scratching my head a bit
here as I have another Samba ADC deployed on another network, similar host
OS, similar RAID setup, and it has been rock solid...

I think the best course of action for this specific situation short-term is
to restore one of the samba backups to get their domain up and running
ASAP.  I'll then have to figure out how to shuffle resources to get a
second ADC running in a VM and with any luck redundancy will mitigate
future corruption issues.  That being said I'm very keen to understand what
the source of this corruption is, as I mentioned earlier I haven't had any
other issues with data corruption on this host and suspect that mismatched
libraries are a big part of the problem.

So, I need to verify, what is the proper way to remove the unwanted
packages in /usr/lib?  Am I trying to remove the correct packages with
apt?  Can I simply rename the offending files and reboot?

Once I've gotten rid of those files I'll follow the restore procedure and
attempt to get samba running again with last week's backup, that will allow
the client to move forward with end of year accounting work and give me a
chance to figure out how to shuffle resources so I can implement the dual
VM architecture.

Please advise, I really need to get this domain functional again by end of
day, I've got about 5 hours to do so.

Thanks again everyone for all of your help.

JS
On Mar 3, 2016 11:30 AM, "mathias dufresne" <infractory at gmail.com> wrote:

>
>
> 2016-03-03 16:32 GMT+01:00 IT Admin <it at cliffbells.com>:
>
>> Apt and to think those packages aren't installed:
>>
>> sudo apt-get remove libtdb-dev libtalloc-dev python-talloc-dev
>> Reading package lists... Done
>> Building dependency tree
>> Reading state information... Done
>> Package 'libtalloc-dev' is not installed, so not removed
>> Package 'libtdb-dev' is not installed, so not removed
>> Package 'python-talloc-dev' is not installed, so not removed
>> The following packages were automatically installed and are no longer
>> required:
>>   linux-image-4.2.0-27-generic linux-image-extra-4.2.0-27-generic
>> Use 'apt-get autoremove' to remove them.
>> 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
>>
>> Should I just rename the files?
>>
>> If the database is corrupt my only recourse is to follow the restore
>> guidelines in the Samba wiki and roll back to a previous version of the
>> db,
>> correct?
>>
>> JS
>>
>
> I'd say that depend on how complex is that corruption and how much you
> have around you to understand that corruption.
>
> If it is too complex (main reason to be too complex is a boss in a hurry)
> restore the whole thing.
>
> During restoration what you need is the same kind of system, you don't
> really need to restore on the very same system. What I mean is you can take
> advantage of that issue to deploy a new system (using same version of that
> system, ie if it was centos6, use centos6).
> As you seem to have a big system doing lot of things, part of these things
> is running a Samba and even that software is configured to do several
> things (AD + file server) which is not advised, I would use that down time
> to switch Samba from that big system to some virtual machine with
> minimalistic system doing one and only one thing: Samba as AD. Then in a
> second VM I would install file server. In fact before installing file
> server I would create another VM to host a second DC.
>
> AD with one DC is not advised, anywhere. If you were having several DC
> perhaps you would not have the whole AD broken but only one DC broken and
> the other(s) one(s) working well. This was already explained to you today,
> I was just insisting a little bit : )
>
>
>
>
>> On Mar 3, 2016 7:40 AM, "Sketch" <smblist at rednsx.org> wrote:
>>
>> > I'd remove the distro packages providing those libs in /usr/lib, as they
>> > could possibly cause problems.  One more I forgot, which might possibly
>> be
>> > responsible for corrupting your ldb database if the wrong one is loaded
>> by
>> > samba at runtime, is libldb.
>> >
>> > From the log, I'm guessing your database is corrupt if it can't read the
>> > schema, but someone else might have more insight.
>> >
>> > On Wed, 2 Mar 2016, IT Admin wrote:
>> >
>> > pytalloc:
>> >>
>> >> /usr/lib/x86_64-linux-gnu/libpytalloc-util.so.2
>> >> /usr/lib/x86_64-linux-gnu/libpytalloc-util.so.2.1.2
>> >> /usr/local/samba/include/pytalloc.h
>> >> /usr/local/samba/lib/private/libpytalloc-util.so.2
>> >> /usr/local/samba/lib/private/libpytalloc-util.so.2.1.3
>> >>
>> >> libtalloc:
>> >>
>> >> /usr/lib/x86_64-linux-gnu/libtalloc.so.2
>> >> /usr/lib/x86_64-linux-gnu/libtalloc.so.2.1.2
>> >> /usr/local/samba/lib/private/libtalloc-report-samba4.so
>> >> /usr/local/samba/lib/private/libtalloc.so.2
>> >> /usr/local/samba/lib/private/libtalloc.so.2.1.3
>> >>
>> >>
>> >> libtdb:
>> >>
>> >> /usr/lib/x86_64-linux-gnu/libtdb.so.1
>> >> /usr/lib/x86_64-linux-gnu/libtdb.so.1.3.5
>> >> /usr/lib/x86_64-linux-gnu/samba/libtdb-wrap.so.0
>> >> /usr/lib/x86_64-linux-gnu/samba/libtdb_compat.so.0
>> >> /usr/local/samba/lib/private/libtdb-wrap-samba4.so
>> >> /usr/local/samba/lib/private/libtdb.so.1
>> >> /usr/local/samba/lib/private/libtdb.so.1.3.7
>> >>
>> >>
>> >> I also noticed that I've got a log.smbd in /usr/local/samba/var:
>> >> http://www.anonpaste.net/?p=bdfa3
>> >>
>> >> JS
>> >>
>> >>
>> >>
>> >> On Wed, Mar 2, 2016 at 9:00 PM, Sketch <smblist at rednsx.org> wrote:
>> >>
>> >> Also check for pytalloc|libtalloc and libtdb.
>> >>>
>> >>>
>> >>> On Wed, 2 Mar 2016, IT Admin wrote:
>> >>>
>> >>> I poked around the system using locate, I don't think there are any
>> >>>
>> >>>> packages installed on this system other than those associated with
>> the
>> >>>> 4.3.3 build I compiled.
>> >>>>
>> >>>> smbd
>> >>>>
>> >>>> /usr/lib/python2.7/dist-packages/samba/samba3/smbd.so
>> >>>> /usr/lib/x86_64-linux-gnu/samba/libsmbd_base.so.0
>> >>>> /usr/lib/x86_64-linux-gnu/samba/libsmbd_conn.so.0
>> >>>> /usr/lib/x86_64-linux-gnu/samba/libsmbd_shim.so.0
>> >>>> /usr/local/samba/lib/private/libsmbd-base-samba4.so
>> >>>> /usr/local/samba/lib/private/libsmbd-conn-samba4.so
>> >>>> /usr/local/samba/lib/private/libsmbd-shim-samba4.so
>> >>>> /usr/local/samba/lib/python2.7/site-packages/samba/samba3/smbd.so
>> >>>> /usr/local/samba/private/smbd.tmp
>> >>>> /usr/local/samba/sbin/smbd
>> >>>> /usr/local/samba/share/man/man8/smbd.8
>> >>>> /usr/local/samba/var/log.smbd
>> >>>> /usr/local/samba/var/log.smbd.old
>> >>>> /usr/local/samba/var/run/smbd.pid
>> >>>> /var/log/upstart/smbd.log.1.gz
>> >>>>
>> >>>> nmbd
>> >>>>
>> >>>> /usr/local/samba/sbin/nmbd
>> >>>> /usr/local/samba/share/man/man8/nmbd.8
>> >>>> /var/log/upstart/nmbd.log.1.gz
>> >>>>
>> >>>> samba
>> >>>>
>> >>>> /usr/local/samba/sbin/nmbd
>> >>>> /usr/local/samba/sbin/samba
>> >>>> /usr/local/samba/sbin/samba_dnsupdate
>> >>>> /usr/local/samba/sbin/samba_kcc
>> >>>> /usr/local/samba/sbin/samba_spnupdate
>> >>>> /usr/local/samba/sbin/samba_upgradedns
>> >>>> /usr/local/samba/sbin/smbd
>> >>>> /usr/local/samba/sbin/winbind
>> >>>>
>> >>>>
>> >>>> If I'm overlooking something obvious please let me know.
>> >>>>
>> >>>> JS
>> >>>>
>> >>>> On Wed, Mar 2, 2016 at 5:42 PM, Marc Muehlfeld <mmuehlfeld at samba.org
>> >
>> >>>> wrote:
>> >>>>
>> >>>> Am 02.03.2016 um 18:52 schrieb IT Admin:
>> >>>>
>> >>>>>
>> >>>>> Samba is compiled from source.
>> >>>>>> Samba DB is stored on local RAID array.
>> >>>>>> Changes to AD are done using ADUC from a Windows 7 box.
>> >>>>>> AD is used for authentication, user shares (folder redirection),
>> and
>> >>>>>>
>> >>>>>> shared
>> >>>>>
>> >>>>> folders.
>> >>>>>>
>> >>>>>>
>> >>>>> Can you add
>> >>>>>   log level = 10
>> >>>>> to your smb.conf, empty your log directory and start Samba. It
>> should
>> >>>>> generate a new log, that captures all output. Then put it to
>> >>>>> cpaste.org
>> >>>>> or some other paste service and share the link with us. Maybe we see
>> >>>>> something interesting.
>> >>>>>
>> >>>>>
>> >>>>> Does something changed when the problem occured the first time? Some
>> >>>>> package updates, crashes, etc.?
>> >>>>>
>> >>>>>
>> >>>>> Can you make sure that no kind of Samba package (daemon, libs,
>> etc.) is
>> >>>>> installed on the system? Maybe your selfcompiled version overwrites
>> >>>>> some
>> >>>>> stuff and your OS installed an update, that mixes now with the self
>> >>>>> compiled version. Just a guess.
>> >>>>>
>> >>>>> Regards,
>> >>>>> Marc
>> >>>>>
>>
>