Samba4 index errors, and upgrade challenges

Sina Owolabi oowolabi at qrios.com
Wed Aug 25 13:57:38 MDT 2010 

As ordered:

../bin/ldbsearch -H sam.ldb -b "CN=Certificate Service DCOM
Access,CN=Users, DC=qrios,DC=com"
# returned 0 records
# 0 entries
# 0 referrals

[root at dc private]# ../bin/ldbsearch -H sam.ldb.d/DC\=QRIOS\,DC\=COM.ldb
"CN=Certificate Service DCOM Access,CN=Users, DC=qrios,DC=com"
# returned 0 records
# 0 entries
# 0 referrals

Not found.

On Wed, 2010-08-25 at 23:42 +0400, Matthieu Patou wrote:
> 
> "Sina Owolabi" <oowolabi at qrios.com> wrote:
> 
> >Yes Mat.
> >I found CN=Certificate Service DCOM Access,CN=Users, DC=qrios,DC=com and
> >i deleted every record that has it in the DC=QRIOS,DC=COM.ldb file (Um,
> >should I have done that?).
> The answer is in the question ....
> >Still fails though, with the same error messages. What do you think
> >needs to be done?
> >Output:
> >=============================================================================
> >Creating a reference provision
> >pdc_fsmo_init: no domain object present: (skip loading of domain
> >details)
> >
> >naming_fsmo_init: no partitions dn present: (skip loading of naming
> >contexts details)
> >
> >naming_fsmo_init: no partitions dn present: (skip loading of naming
> >contexts details)
> >
> >naming_fsmo_init: no partitions dn present: (skip loading of naming
> >contexts details)
> >
> >Copy privilege
> >Update base samdb by searching difference with reference one
> >Starting update of samdb
> >There are 143 missing objects
> >CN=Certificate Service DCOM Access,CN=Users, DC=qrios,DC=com
> >Traceback (most recent call last):
> >  File "./scripting/bin/upgradeprovision", line 1580, in ?
> >    schema):
> >  File "./scripting/bin/upgradeprovision", line 1225, in update_samdb
> >    schema, highestUSN)
> >  File "./scripting/bin/upgradeprovision", line 1002, in
> >update_partition
> >    add_missing_entries(ref_samdb, samdb, names, basedn, listMissing)
> >  File "./scripting/bin/upgradeprovision", line 674, in
> >add_missing_entries
> >    hashMissing, index)
> >  File "./scripting/bin/upgradeprovision", line 535, in
> >add_missing_object
> >    if handle_special_add(samdb, dn, names):
> >  File "./scripting/bin/upgradeprovision", line 487, in
> >handle_special_add
> >    samdb.delete(res[0]["dn"])
> >_ldb.LdbError: (16, 'No such attribute (16)')
> 
> Obviously this means that it is still here. Can you send the output of the ldbsearch for this object both on the Sam.ldb and on the ldb starting with dc= ....
> >A transaction is still active in ldb context [0x69eb640]
> >on /usr/local/samba/private/sam.ldb
> >A transaction is still active in ldb context [0x7cc5c40]
> >on /usr/local/samba/private/idmap.ldb
> >A transaction is still active in ldb context [0x7f32620]
> >on /usr/local/samba/private/secrets.ldb
> >A transaction is still active in ldb context [0x7cc21a0]
> >on /usr/local/samba/private/privilege.ldb
> >A transaction is still active in ldb context [0x8e1dcb0]
> >on /usr/local/samba/private/referenceprovisionISkpS_/private/sam.ldb
> >A transaction is still active in ldb context [0x9195d40]
> >on /usr/local/samba/private/referenceprovisionISkpS_/private/idmap.ldb
> >A transaction is still active in ldb context [0x8867010]
> >on /usr/local/samba/private/referenceprovisionISkpS_/private/secrets.ldb
> >A transaction is still active in ldb context [0x711e280]
> >on /usr/local/samba/private/referenceprovisionISkpS_/private/privilege.ldb
> >=============================================================================
> >
> >
> >
> >
> >
> >
> >On Wed, 2010-08-25 at 10:04 +0400, Matthieu Patou wrote:
> >> On 24/08/2010 23:40, Sina Owolabi wrote:
> >> > Hi Matthieu.
> >> > Below are the logs from another failed upgradeprovision from alpha9 to
> >> > alpha12. We took a git pull last on August 5th, or late June.
> >> > What do you think is the problem?
> >> > Thanks for all your help!
> >> >
> >> > [root at dc source4]# ./scripting/bin/upgradeprovision --full
> >> > -s /usr/local/samba/etc/smb.conf
> >> > Creating a reference provision
> >> > pdc_fsmo_init: no domain object present: (skip loading of domain
> >> > details)
> >> >
> >> > naming_fsmo_init: no partitions dn present: (skip loading of naming
> >> > contexts details)
> >> >
> >> > naming_fsmo_init: no partitions dn present: (skip loading of naming
> >> > contexts details)
> >> >
> >> > naming_fsmo_init: no partitions dn present: (skip loading of naming
> >> > contexts details)
> >> >
> >> > Copy privilege
> >> > Update base samdb by searching difference with reference one
> >> > Starting update of samdb
> >> > There are 143 missing objects
> >> > CN=Certificate Service DCOM Access,CN=Users, DC=qrios,DC=com
> >> 
> >> can you do this: ldbsearch -H <path_to_samba>/private/sam.ldb -b 
> >> "CN=Certificate Service DCOM Access,CN=Users, DC=qrios,DC=com"
> >> 
> >> it's best if you can do this with a alpha9 ldbsearch.
> >> 
> >> I can remember that some old attributes are blocking the upgrade, we 
> >> have to find them and remove them.
> >> 
> >> 
> >> > Traceback (most recent call last):
> >> >    File "./scripting/bin/upgradeprovision", line 1580, in ?
> >> >      schema):
> >> >    File "./scripting/bin/upgradeprovision", line 1225, in update_samdb
> >> >      schema, highestUSN)
> >> >    File "./scripting/bin/upgradeprovision", line 1002, in
> >> > update_partition
> >> >      add_missing_entries(ref_samdb, samdb, names, basedn, listMissing)
> >> >    File "./scripting/bin/upgradeprovision", line 674, in
> >> > add_missing_entries
> >> >      hashMissing, index)
> >> >    File "./scripting/bin/upgradeprovision", line 535, in
> >> > add_missing_object
> >> > if handle_special_add(samdb, dn, names):
> >> >    File "./scripting/bin/upgradeprovision", line 487, in
> >> > handle_special_add
> >> >      samdb.delete(res[0]["dn"])
> >> > _ldb.LdbError: (16, 'No such attribute (16)')
> >> > A transaction is still active in ldb context [0x45179a0]
> >> > on /usr/local/samba/private/sam.ldb
> >> > A transaction is still active in ldb context [0x4af2830]
> >> > on /usr/local/samba/private/idmap.ldb
> >> > A transaction is still active in ldb context [0x5a6a6c0]
> >> > on /usr/local/samba/private/secrets.ldb
> >> > A transaction is still active in ldb context [0x55fd850]
> >> > on /usr/local/samba/private/privilege.ldb
> >> > A transaction is still active in ldb context [0x6d54780]
> >> > on /usr/local/samba/private/referenceprovision7dOw3-/private/sam.ldb
> >> > A transaction is still active in ldb context [0x572c6d0]
> >> > on /usr/local/samba/private/referenceprovision7dOw3-/private/idmap.ldb
> >> > A transaction is still active in ldb context [0x554ca40]
> >> > on /usr/local/samba/private/referenceprovision7dOw3-/private/secrets.ldb
> >> > A transaction is still active in ldb context [0x4b576a0]
> >> > on /usr/local/samba/private/referenceprovision7dOw3-/private/privilege.ldb
> >> > [root at dc source4]#
> >> >
> >> > On Sat, 2010-08-21 at 11:34 +0400, Matthieu Patou wrote:
> >> >> On 21/08/2010 11:20, oowolabi at qrios.com wrote:
> >> >>> Thank you so very much Matthieu. I guess the answer was right under our noses all the time!
> >> >>> We edited the .ldb files as you advised, but it didn't work until we tracked and deleted down every instance of the problematic CN's index and data(the CN showing up in the log snippet earlier). We believe power outage on samba4-alpha9 plus a faulty user creation caused this.
> >> >> Ok
> >> >>> We've tried our hands at upgradeprovision, but it didn't work. What we did: copied an existing samba4-alpha12 samba-master folder to the machine, and ran upgradeprovision as you suggested (and as we saw in the help file included in the source4 folder). It appeared to upgrade, but we believe parts of the db didn't upgrade. We had SPNEGO errors when trying to use ADUC to connect to the samba4 instance, and 'unknown username&   password' on the WinXP when trying to connect. Unfortunately we are unable to put the logs here right now.  We attempted a second time with upgradeprovision, with '--full' but it complains about uncompleted ldb transactions.
> >> >> Show me the errors, for your alpha9 you need the --full options, also I
> >> >> corrected a lot of problems in upgradeprovision in june so is your test
> >> >> prior this date ?
> >> >>
> >> >>> We ended up reinstalling the alpha9 and replacing etc, private and var directories from backup before it would restart and function properly.
> >> >>> We intend to attempt it again this weekend. But then, do you have any idea why that happened? So sorry about not being able to attach logs, this email is being
> >> >>> composed on a handheld.
> >> >> Ok send the log if you can !
> >> >> Matthieu.
> >> >>> Thank you again, very, very much, for your help, and we hope to hear from you again!
> >> >>>
> >> >>> very best regards,
> >> >>>
> >> >>> Qrios
> >> >>> Sent from my BlackBerry wireless device from MTN
> >> >>>
> >> >>> -----Original Message-----
> >> >>> From: Matthieu Patou<mat at samba.org>
> >> >>> Date: Fri, 20 Aug 2010 01:06:13
> >> >>> To:<oowolabi at qrios.com>
> >> >>> Reply-To: mat at samba.org
> >> >>> Cc: Mosebolatan Adetoro<madetoro at qrios.com>; Stefan Metzmacher<metze at SerNet.DE>;<samba-technical at samba.org>; Johannes Loxen<jl at sernet.de>;<samba at SerNet.DE>
> >> >>> Subject: Re: Samba4 index errors, and upgrade challenges
> >> >>>
> >> >>>     On 19/08/2010 16:50, oowolabi at qrios.com wrote:
> >> >>>> Hi, Matthieu.
> >> >>>>
> >> >>>> We at Qrios were referred to you by Stefan Metzmacher regarding our issues with samba-4(alpha-9), running on a RHEL 5.3 64-bit server, which we have currently deployed for a friendly company willing to try out open source domain services, in lieu of AD.
> >> >>>> In a nutshell, ADUC (and all other ldap browser tools we have tried to utilize) complains 'an operational error has occurred' (classic MSFT empty error message!), and is unable to enumerate the objects and directories in the domain. Strangely enough, most domain user objects are still searchable and modifiable to a large extent. (It cannot search and find all of them, though. )
> >> >>>> On viewing the samba logs, we see this little snippet when ADUC (and other tools) attempt to browse the tree:
> >> >>>>
> >> >>>> [Thu Aug 12 20:59:41 2010 WAT, 1 lib/ldb_wrap.c:68:ldb_wrap_debug()]
> >> >>>>> ldb: Invalid data for index CN=Esther O. Tewogbola,DC=skyebankzm,DC=net
> >> >>>> We have found several .ldb files in the /usr/local/samba/private/sam.ldb.d/ directory and discovered which one holds the errant index record. We have tried to delete the index and related ones (this index is for a user that was created badly) using samba-4's tdbtool (after exhaustively searching for any tool that can modify an .ldb file. Deleting the index does not solve the problem.
> >> >>> Well it's highly recommended to use ldbedit/ldbsearch/ldbmodify to
> >> >>> modify ldb files and it's __very__ recommended to modify them under the
> >> >>> control of the samdb (that is to say do something like ldbedit -H
> >> >>> private/sam.ldb rather than ldbedit -H private/sam.ldb.d/DC=sambaorg,
> >> >>> DC=corp.ldb).
> >> >>>> Stefan informs us you have had such index problems in the last few days and you have been successful in solving them. Can you please share with us what you have been able to do, so we can sanitize the database?
> >> >>> Well I had a couple of index pb last week but they were due to the fact
> >> >>> that ldb wanted to reindex my provision after upgrade.
> >> >>> I made the following patch 2651c2f98841a3521b6893ae5158bbb81832b7ee in
> >> >>> my upgradeprovsion-wip branch on
> >> >>> http://gitweb.samba.org/?p=mat/samba.git;a=shortlog;h=refs/heads/upgradeprovision-wip.
> >> >>>
> >> >>> But I'm pretty sure it won't work for you. My advice is to trash the
> >> >>> index and to force ldb to recreate it.
> >> >>>
> >> >>> If I were you here is what I would do:
> >> >>>
> >> >>> 1) Stop samba
> >> >>> 2) Take a backup (or 2) of the samba provision
> >> >>> 3) copy 1 backup somewhere else and modify the smb.conf to point to the
> >> >>> folder. Ie if you put the provision in /usr/local/backupprovision, the
> >> >>> file /usr/local/backupprovision/etc/smb.conf must have an entry private
> >> >>> dir with the following content: /usr/local/backupprovision/private, and
> >> >>> a lock dir with the following value  /usr/local/backupprovision (modify
> >> >>> also the path for the sysvol and netlogon although not mandatory it's
> >> >>> better that everything is coherent)
> >> >>> 4) ldbedit -H /usr/local/backupprovision/private/sam.ldb -o modules:, it
> >> >>> will open the file sam.ldb without loading the modules (otherwise you
> >> >>> have the module loaded and it looks different)
> >> >>> 5) Locate the entries @INDEXLIST remove all the IDXATTR entries, save
> >> >>> and exit, this should force samdb to reindex the whole database
> >> >>> 6) ldbedit -H /usr/local/backupprovision/private/sam.ldb, it will take
> >> >>> some time as ldb is reindexing your provision (it can take up to 20
> >> >>> minutes for a 20 000 users/contacts/computer provision)
> >> >>>
> >> >>> Hopefully it should manage to remove the dirty index and rebuild it. If
> >> >>> not well let me know !
> >> >>> After to check that every thing is ok you have to make a search on the
> >> >>> user with a pb:
> >> >>>
> >> >>> ldbsearch -H /usr/local/backupprovision/private/sam.ldb  -b "CN=Esther
> >> >>> O. Tewogbola,DC=skyebankzm,DC=net"
> >> >>>
> >> >>> If every thing is ok then copy the sam.ldb file and the sam.ldb.d folder
> >> >>> back to the initial place.
> >> >>>> Also, we would like very much to be able to upgrade from alpha-9 to 12, and run samba-4 in at least a replicated (if not completely clustered) mode, in order to accommodate increased connections to the samba-4 service (more users).
> >> >>> I'm not 100% sure I understand your term of replicated/clustered. Do you
> >> >>> speak about file system served by S4 in this case you won't gain much
> >> >>> from using 2 or 3 samba4 servers as it didn't support the clustering
> >> >>> mode (yet) nor ms-dfs for share different from sysvol and netlogon (this
> >> >>> two are working with ms-dfs). If you speak about Directory services, yes
> >> >>> it can help although I'm surprised that you have problems, how many
> >> >>> users are in your AD ? In a normal mode the active directory server is
> >> >>> used with burst in the morning (when everybody log in) and then it is
> >> >>> pretty calm unless users are connecting all day long to tons and tons of
> >> >>> servers (so that it will require a lot of verification for the AD).
> >> >>>
> >> >>> Well in anycase the only good solution is to have replicated DCs
> >> >>>> What we have done in attempting to upgrade was to setup alpha-12 without provisioning, and then rsync -avHk the samba etc and private directories.
> >> >>> As I said the only good solution is to have replicated DCs, here what
> >> >>> you are doing is duplicating the information of the 1st DC so you'll end
> >> >>> with 2 server with the same server information, it's not too great as
> >> >>> password are not replicated and client can get confused.
> >> >>>>     It seems to work, but the logs show that alpha-12 is not altogether happy with that. We've also tried to vampire from 9 to 12, without success(following the howto). Please, what works?
> >> >>> Well show us the log.
> >> >>> Once you fixed your index you can try upgradeprovision with from the git
> >> >>> tree: upgradeprovsion -s /usr/local/backupprovision/etc/smb.conf, test
> >> >>> it with a copy of your provision somewhere else, it should work (I've
> >> >>> been able to upgrade my production which is an alpha3 updated to several
> >> >>> milestone up to alpha9/10).
> >> >>>
> >> >>> Then try to vampire with the help of the howto.
> >> >>>
> >> >>> If needed send email to the samba-technical list or join irc channel on
> >> >>> irc.freenode.org!
> >> >>>
> >> >>> Cheers
> >> >>> Matthieu.
> >> >>>> Hope to hear from you soon.
> >> >>>> Very best regards,
> >> >>>>
> >> >>>> Sina Owolabi
> >> >>>> ------Original Message------
> >> >>>> From: Mosebolatan Adetoro
> >> >>>> To: Stefan Metzmacher
> >> >>>> Cc: Johannes Loxen
> >> >>>> Cc: samba at SerNet.DE
> >> >>>> Cc: oowolabi
> >> >>>> Subject: Re: Invalid data for index error [TT#65245]
> >> >>>> Sent: Aug 19, 2010 7:39 AM
> >> >>>>
> >> >>>> Hi Stefan,
> >> >>>>
> >> >>>> Thanks for this useful information!
> >> >>>>
> >> >>
> >> 
> >> 
> >
> >-- 
> >best regards,
> >
> >
> >Sina Owolabi
> >
> >Tel: +234 709 814 1714  Qrios
> >Mob: +234 803 402 2578
> >Fax: +234 709 814 1716
> >Zimbra Gold Partners
> >Red Hat Certified Training Partners
> >http://www.qrios.com
> >oowolabi at qrios.com
> >27B Adewale Kolawole Crescent, Lekki Peninsula, Oceanside 
> >
> 
> Matthieu Patou
> Samba team
> 

-- 
best regards,


Sina Owolabi

Tel: +234 709 814 1714  Qrios
Mob: +234 803 402 2578
Fax: +234 709 814 1716
Zimbra Gold Partners
Red Hat Certified Training Partners
http://www.qrios.com
oowolabi at qrios.com
27B Adewale Kolawole Crescent, Lekki Peninsula, Oceanside

More information about the samba-technical mailing list