[Samba] Restoring Samba databases from /var/lib/samba

[Sebastian Arcus]

On 01/06/2022 23:15, Andrew Bartlett wrote:
> On Wed, 2022-06-01 at 22:54 +0100, Sebastian Arcus via samba wrote:
>> To start with the end, until today I never realised that there are
>> specific procedures for backing up Samba AD databases - which is my
>> bad.
>> I've always backed up /var/lib/samba and /var/cache/samba, seeing as
>> that's where Samba kept its stuff. Today I've accidentally deleted
>> /var/lib/samba, and tried to copy it back from the nightly backups.
>> Needless to say that it all went to pots, and dns is not working
>> properly any more, not matter what I try. I can provide more details
>> and
>> logs, but first I wanted to ask if it is even worth the effort? Is
>> my
>> backup of /var/lib/samba basically useless to restore things to
>> where
>> they were before?
> 
> samba_upgradedns can fix the links for BIND9_DLZ, but your issues seem
> worse than that.

Indeed, I've already tried it, and it fails:

samba_upgradedns --dns-backend=BIND9_DLZ
Reading domain information
DNS accounts already exist
No zone file /var/lib/samba/bind-dns/dns/redacted.LAN.zone
DNS records will be automatically created
Traceback (most recent call last):
   File "/usr/sbin/samba_upgradedns", line 348, in <module>
     ncname = msg[0]['nCName'][0]
KeyError: 'No such element'


> 
>> Some basic info:
>> OS: Slackware 14.1
>> Samba: 4.9.4
>> Mode: Active Directory DC with file server on the same machine -
>> only
>> one DC on domain
>>
>> Briefly, the samba_dlz plugin seems to be loading, but the logs have
>> various errors which so far I can't make sense of:
>>
>> Jun  1 22:36:05 srv-01-op samba[11769]:
>> ../source4/dsdb/kcc/kcc_periodic.c:768: Failed samba_kcc -
>> NT_STATUS_ACCESS_DENIED
>>
>> and:
>>
>> # samba-tool dns zonelist localhost -U Administrator
>> Password for [redacted\Administrator]:
>> ERROR(runtime): uncaught exception - (9717,
>> 'WERR_DNS_ERROR_DS_UNAVAILABLE')
>>     File "/usr/lib64/python2.7/site-
>> packages/samba/netcmd/__init__.py",
>> line 177, in _run
>>       return self.run(*args, **kwargs)
>>     File "/usr/lib64/python2.7/site-packages/samba/netcmd/dns.py",
>> line
>> 670, in run
>>       request_filter)
> 
> I would look at the server logs more, and things like 'samba-tool
> dbcheck --cross-ncs'

I've already tried that, and it finds a bunch of errors (see below), 
which --fix unfortunately doesn't manage to fix:

samba-tool dbcheck --cross-ncs
Checking 3498 objects
ERROR: NC DC=DOMAINDNSZONES,DC=redacted,DC=LAN lacks a reference to a 
Deleted Objects container
ERROR: NC DC=FORESTDNSZONES,DC=ORIGINPROBATE,DC=LAN lacks a reference to 
a Deleted Objects container
WARNING: no target object found for GUID component for cross-partition 
link msDS-HasInstantiatedNCs in object CN=NTDS 
Settings,CN=SRV-01-OP,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=redacted,DC=lan 
- 
B:8:0000000D:<GUID=36a8fb71-8636-4141-9cee-63728a87b8d5>;<RMD_ADDTIME=131951543060000000>;<RMD_CHANGETIME=131951543060000000>;<RMD_FLAGS=0>;<RMD_INVOCID=5789974e-1b30-4f8d-9b81-d4905423e18f>;<RMD_LOCAL_USN=3719>;<RMD_ORIGINATING_USN=3719>;<RMD_VERSION=1>;DC=ForestDnsZones,DC=redacted,DC=lan
Not removing dangling one-way cross-partition link (we might be 
mid-replication)

The above sequence is repeated a number of times.


> 
>> Should I just cut my loses and rebuild everything from scratch? It
>> will
>> involve work from my part and downtime for the users, but I should
>> have
>> really known about proper Samba AD db backups, so it is what it is.
>>
>> Any pointers much appreciated.
> 
> Our DBs need to be backed up with the locks taken, otherwise you can
> find it mid-modify.  Otherwise it is just pure luck as to if it was
> quiet at the time.

One thing I find a bit strange is that I tried to restore copies of 
/var/lib/samba taken on a number of different days in the past week, and 
they all result in the same corrupt state. I guess the chances of 
getting a clean instance of the databases by just copying the files is 
pretty low.

This domain doesn't have a lot of machines - I guess I'll just have to 
accept my losses and rebuild everything - and put it down to experience. 
Thank you for taking the time to look into it.

Sebastian