TDB corruption on sam.ldb tdb files

[Andrew Bartlett]

On Mon, 2012-09-17 at 12:52 +1200, Andrew Walters wrote:
> I didn't have any luck posting this to the samba mailing list (besides an out-of-office auto-reply!) so thought I'd try here. Sorry for the double-post.

No worries.  I did see it, and it is very serious.  Thanks for posting
it here. 

> Hi all, 
> 
> I've been successfully running Samba4 at two schools I administer as
> well as my home test network, starting with alpha17 and now on beta8.
> Group Policy works a charm. 
> 
> But lately at one of the schools it seems sam.ldb has got messed up. 

Indeed.  As I see further down, the corruption is very, very bad. 

> In ADUC, I can browse all users, groups and machines fine, everything
> and everyone authenticates and operates fine, and I can view all
> current group memberships for a user and existing users in a group,
> but if searching for users to add to a group, or to add groups to a
> user, only a small subset of users/groups (respectively) shows in the
> search results, and if I type the name of any other user/group
> manually, I get told by ADUC that they do not exist. I only see 6 out
> of about 90 users, and 22 out of about 65 groups (all the builtins
> seem to show as part of that 22). 
> 
> This means my AD is more or less stuck from an administrative point of
> view. I can generally not change user or group memberships without
> difficulty. 
> This looks like it happened while I was on leave for a few weeks, so
> backups of non-corrupt data have been overwritten - I only had a
> two-week rotation/retention policy on /srv/adsrv/var contents (changed
> since!). 
> 
> So in ADUC I can view group members or view user groups but can't
> modify the bulk of them. 
> 
> samba-tool behaves the opposite - "samba-tool group listmembers
> (groupname)" only lists users if they're in the same set of 6, but
> addmember succeeds - if I use addmember, while listmembers still
> doesn't show the newly added member to a group, opening the group in
> ADUC does list the member. 
> 
> I can't discern any pattern or common element exclusive to those 6
> users. 
> 
> If I do a 'ldbsearch -H sam.ldb "objectClass=*"', out of the user
> records returned, only the same 6 that show up in AD searches show up
> in the results (amongst other machine and non-user objects). 
> 
> samba-tool dbcheck --cross-ncs returns "Checked 3229 objects (0
> errors)", but samba-tool dbcheck --reindex fails with: 
> 
> =========================== 
> Re-indexing... 
> Invalid data for index
> DC=_kerberos._tcp.Default-First-Site-Name._sites.dc,DC=_msdcs.ad.(domain name),CN=MicrosoftDNS,DC=ForestDnsZones,DC=ad,DC=(domain name) 
> 
> ltdb: tdb(/srv/adsrv/var/lib/samba/private/sam.ldb.d/DC=AD,DC=(domain
> name).ldb): tdb_rec_read bad magic 0x6863733d at offset=1773572 
> 
> re-indexed database : (1, "attribute 'force_reindex': no matching
> attribute value while deleting attribute on '@ATTRIBUTES'") 
> =========================== 

This is about as serious as you can get.  Essentially you need to shut
the whole thing down, and back it up.  tdbbackup as suggested would be a
good idea if the databases were able to be read, but this is not the
case - it is corrupt. 

Once copied in a cold backup, then test tdbbackup on each file.  If
tdbbackup does work, then this is a good sign, as we can then restore at
least some data from that, and then re-index without the previous,
corrupt index values.  Give me (in private if required) the logs, so we
can try and tell how bad the damage is. 

> (I have the samba4 tree contained in /srv/adsrv on this server to
> isolate it from a samba 3 instance doing the file sharing, inspired by
> "Franky" - this is left over from a configuration to suit alpha17 (the
> smbd subprocess didn't work back then for shares) and otherwise works
> fine, also works fine at the other school). 
> 
> I can't browse past the
> Default-First-Site-Name._sites.dc,DC=_msdcs.ad.(domain
> name),CN=MicrosoftDNS,DC=ForestDnsZones,DC=ad,DC=(domain name) folder
> using the Windows-based LDAP_Admin.exe utility, it throws this error: 
> "LDAP error! Operations Error: 00002020: schema: metadata tdb not
> initialized at ../source4/dsdb/samdb/ldb_modules/schema_load.c:117" 
> 
> Based on the advice here: 
> 
> http://lists.samba.org/archive/samba-technical/2010-December/075239.html 
> 
> ... I tried to manually remove the index by doing this: 
> 
> /srv/adsrv/bin/ldbedit -H /srv/adsrv/var/lib/samba/private/sam.ldb -s
> base -b \@INDEXLIST 
> 
> ... and clearing out the index to the example given in the above link.
> Or even just removing one entry. However, any modifications fail with
> a similar error to the above reindex command: 
> 
> =========================== 
> ltdb: tdb(/srv/adsrv/var/lib/samba/private/sam.ldb.d/DC=AD,DC(domain
> name).ldb): tdb_rec_read bad magic 0x6863733d at offset=1773572 
> 
> failed to modify @INDEXLIST - ldb_wait: Operations error (1) 
> =========================== 
> ... and the modification doesn't happen. 
> 
> Argh! 

The dbcheck command you tried does (essentially) this, so it is expected
that they are equivalent.  

> Any ides as to how I may be able to get out of this? Any help
> appreciated. 

Let's see if we can somehow get most of your data back.  I'm hoping some
of my fellow team mates who have needed to recover other tdb databases
may also be able to help or give advise from their experiences.

Thanks,

Andrew Bartlett

-- 
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org