facing very slow authentication responses from samba4, urget help needed

[Andreas Oster]

Am 14.05.2012 16:00, schrieb Matthieu Patou:
> On 05/14/2012 01:53 AM, Andreas Oster wrote:
>> Am 12.05.2012 10:21, schrieb Matthieu Patou:
>>> Hello Andreas
>>>> Hello Matthieu,
>>>>
>>>> thanks for the fat reply.
>>>>
>>>> We have about the same user number.I have migrated our old win2000
>>>> AD to
>>>> samba4 and raised forest and domain level to 2008r2. In my test (esx)
>>>> environment I have only tested with some clients (WinXP, WinVista, 2k8
>>>> server) and it seemed to work well, at least authentication, GPOs and
>>>> sripts did work like before. I did not recognize such delays in my
>>>> testing environment. Unfortunately I did not test the LDAP stuff.
>>>>
>>>> Any idea how to find out where the delay is generated ? What I have
>>>> observed is that sometimes the samba processes use up a lot of CPU.
>>> Ok, the reason might be indexing because I doubt you upgraded the schema
>>> so it's not a true forest raise to 2008r2 (my guess).
>>>
>>> Related to search that should exploit indexes that should have been
>>> present you can also have unindexed search.
>>>
>>> You can try to debug it by activating the warnings on unindexed search,
>>> to do so you have to set LDB_WARN_UNINDEXED to 1. For instance you could
>>> start samba this way (log level has to be 2 or more ...)
>>>
>>> sudo -c sh "LDB_WARN_UNINDEXED=1 ./bin/samba -d 2"
>>>
>>> Then any unindexed search will produce the following output:
>>>
>>> ldb: ldb FULL SEARCH: (&(!(isDeleted=TRUE))(msds-intid=*)) SCOPE: sub
>>> DN: DC=exchange,DC=home,DC=matws,DC=net
>>>
>>> In this example I'm doing a search on msds-intid which is not an indexed
>>> attribute so it's expected that I'll have a full scan.
>>>
>>> You can test also indexed search directly on the command line with ldb
>>> by querying the database directly:
>>>
>>> LDB_WARN_UNINDEXED=1 ./bin/ldbsearch -H
>>> ~/workspace/samba/rodc_mat/private/sam.ldb --cross-ncs '(msds-intid=*)'
>>>
>>> If you don't yield interesting results, you can start to trace the ldap
>>> traffic with wireshark and see how long LDAP requests take. If the
>>> request takes more than 0.1 second you should have a look (if the
>>> request is frequently executed). With wireshark and adapted
>>> keytab/certificates you'll be able to get the clear text of the query.
>>>
>>> Once you have it you can trace it with callgrind:
>>> sudo valgrind --tool=callgrind  ./bin/ldbsearch -H path_to_your_sam.ldb
>>> -U user%password '(my_slow_ldap_query)'
>>>
>>> kcachegrind will help you to analyze it.
>>>
>>>
>>> Here are two pointers for capturing packets:
>>> https://wiki.samba.org/index.php/Capture_Packets
>>> https://wiki.samba.org/index.php/Analyze_Capture
>>>
>>>
>>> Matthieu
>>>
>> Hello Matthieu,
>>
>> actually I did a domain and forest raise to 2008r2 before migrating to
>> samba4.
>>
>> I have tried with ldbsearch but am not sure if I did it in the correct
>> way:
>>
>> LDB_WARN_UNINDEXED=1 ./ldbsearch -H ../private/sam.ldb --cross-ncs
>> '(msds-intid=*)' -Uadministrator
>>
>> ldb FULL SEARCH: (&(!(isDeleted=TRUE))(msds-intid=*)) SCOPE: UNKNOWN DN:
>>
>> ldb FULL SEARCH: (&(!(isDeleted=TRUE))(msds-intid=*)) SCOPE: UNKNOWN DN:
>> CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>>
>> ldb FULL SEARCH: (&(!(isDeleted=TRUE))(msds-intid=*)) SCOPE: UNKNOWN DN:
>> CN=Configuration,DC=novanetwork,DC=loc
>>
>> ldb FULL SEARCH: (&(!(isDeleted=TRUE))(msds-intid=*)) SCOPE: UNKNOWN DN:
>> DC=novanetwork,DC=loc
> Ok you you did the search but here you replicated my search that is
> unindexed and was for the purpose of the demonstration.
> What you need to do is to start samba with the LDB_WARN_UNINDEXED=1
> environment var set and a log level of 2 or more.
> With this parameter you should see all the unindexed search that samba
> is doing and that might be causing its slowlyness.
> 
> Matthieu.
> 
> 
Hello Matthieu,

with your help I was able to find one cause of the problem. It seems
that my postfix configuration is faulty. When sending a mail several
different ldap queries are sent to the samba server. This has not been
an issue when using the Windows DC as it could handle the requests much
faster than the samba4 server. These queries seem to put a lot of stress
to the samba processes, as the CPU utilization gets about 100% in those
situations (2cores >3Ghz). I will try to fix my postfix configuration to
remove the useless LDAP queries.

Thank you for your kind help

best regards

Andreas