facing very slow authentication responses from samba4, urget help needed

[Matthieu Patou]

On 05/14/2012 11:16 AM, Andreas Oster wrote:
> Am 14.05.2012 16:00, schrieb Matthieu Patou:
>> On 05/14/2012 01:53 AM, Andreas Oster wrote:
>>> Am 12.05.2012 10:21, schrieb Matthieu Patou:
>>>> Hello Andreas
>>>>> Hello Matthieu,
>>>>>
>>>>> thanks for the fat reply.
>>>>>
>>>>> We have about the same user number.I have migrated our old win2000
>>>>> AD to
>>>>> samba4 and raised forest and domain level to 2008r2. In my test (esx)
>>>>> environment I have only tested with some clients (WinXP, WinVista, 2k8
>>>>> server) and it seemed to work well, at least authentication, GPOs and
>>>>> sripts did work like before. I did not recognize such delays in my
>>>>> testing environment. Unfortunately I did not test the LDAP stuff.
>>>>>
>>>>> Any idea how to find out where the delay is generated ? What I have
>>>>> observed is that sometimes the samba processes use up a lot of CPU.
>>>> Ok, the reason might be indexing because I doubt you upgraded the schema
>>>> so it's not a true forest raise to 2008r2 (my guess).
>>>>
>>>> Related to search that should exploit indexes that should have been
>>>> present you can also have unindexed search.
>>>>
>>>> You can try to debug it by activating the warnings on unindexed search,
>>>> to do so you have to set LDB_WARN_UNINDEXED to 1. For instance you could
>>>> start samba this way (log level has to be 2 or more ...)
>>>>
>>>> sudo -c sh "LDB_WARN_UNINDEXED=1 ./bin/samba -d 2"
>>>>
>>>> Then any unindexed search will produce the following output:
>>>>
>>>> ldb: ldb FULL SEARCH: (&(!(isDeleted=TRUE))(msds-intid=*)) SCOPE: sub
>>>> DN: DC=exchange,DC=home,DC=matws,DC=net
>>>>
>>>> In this example I'm doing a search on msds-intid which is not an indexed
>>>> attribute so it's expected that I'll have a full scan.
>>>>
>>>> You can test also indexed search directly on the command line with ldb
>>>> by querying the database directly:
>>>>
>>>> LDB_WARN_UNINDEXED=1 ./bin/ldbsearch -H
>>>> ~/workspace/samba/rodc_mat/private/sam.ldb --cross-ncs '(msds-intid=*)'
>>>>
>>>> If you don't yield interesting results, you can start to trace the ldap
>>>> traffic with wireshark and see how long LDAP requests take. If the
>>>> request takes more than 0.1 second you should have a look (if the
>>>> request is frequently executed). With wireshark and adapted
>>>> keytab/certificates you'll be able to get the clear text of the query.
>>>>
>>>> Once you have it you can trace it with callgrind:
>>>> sudo valgrind --tool=callgrind  ./bin/ldbsearch -H path_to_your_sam.ldb
>>>> -U user%password '(my_slow_ldap_query)'
>>>>
>>>> kcachegrind will help you to analyze it.
>>>>
>>>>
>>>> Here are two pointers for capturing packets:
>>>> https://wiki.samba.org/index.php/Capture_Packets
>>>> https://wiki.samba.org/index.php/Analyze_Capture
>>>>
>>>>
>>>> Matthieu
>>>>
>>> Hello Matthieu,
>>>
>>> actually I did a domain and forest raise to 2008r2 before migrating to
>>> samba4.
>>>
>>> I have tried with ldbsearch but am not sure if I did it in the correct
>>> way:
>>>
>>> LDB_WARN_UNINDEXED=1 ./ldbsearch -H ../private/sam.ldb --cross-ncs
>>> '(msds-intid=*)' -Uadministrator
>>>
>>> ldb FULL SEARCH: (&(!(isDeleted=TRUE))(msds-intid=*)) SCOPE: UNKNOWN DN:
>>>
>>> ldb FULL SEARCH: (&(!(isDeleted=TRUE))(msds-intid=*)) SCOPE: UNKNOWN DN:
>>> CN=Schema,CN=Configuration,DC=novanetwork,DC=loc
>>>
>>> ldb FULL SEARCH: (&(!(isDeleted=TRUE))(msds-intid=*)) SCOPE: UNKNOWN DN:
>>> CN=Configuration,DC=novanetwork,DC=loc
>>>
>>> ldb FULL SEARCH: (&(!(isDeleted=TRUE))(msds-intid=*)) SCOPE: UNKNOWN DN:
>>> DC=novanetwork,DC=loc
>> Ok you you did the search but here you replicated my search that is
>> unindexed and was for the purpose of the demonstration.
>> What you need to do is to start samba with the LDB_WARN_UNINDEXED=1
>> environment var set and a log level of 2 or more.
>> With this parameter you should see all the unindexed search that samba
>> is doing and that might be causing its slowlyness.
>>
>> Matthieu.
>>
>>
> Hello Matthieu,
>
> with your help I was able to find one cause of the problem. It seems
> that my postfix configuration is faulty. When sending a mail several
> different ldap queries are sent to the samba server. This has not been
> an issue when using the Windows DC as it could handle the requests much
> faster than the samba4 server. These queries seem to put a lot of stress
> to the samba processes, as the CPU utilization gets about 100% in those
> situations (2cores>3Ghz).
Well part of the reason is that the LDAP is only done by one process at 
most so you can only saturate 1 core, where I suppose that MS AD DC are 
able to split in multiple threads.
Please also note that we have plans to improve the speed of Samba we 
definitely know that we are not very good in some area in the AD database.

I would be interested if you could share the query that were faulty in 
the long run it can be instructive to see what we can do to solve this.
> I will try to fix my postfix configuration to
> remove the useless LDAP queries.
Matthieu.


-- 
Matthieu Patou
Samba Team
http://samba.org