[Samba] winbindd stays at 100% CPU having joined a domain

[Thomas Adam]

[ Please would anyone replying to this maintain the Cc list here as
neither myself or my dear colleague are subscribed to this mailing list. ]

Hello all,

I am trying to track down an oddity with winbindd.  We're using samba only
to join a given domain -- so the config file is very minimal, as per:

 [global]
 workgroup = TEDDYBEARS
 netbios name = SMOOTHWALL
 realm = TEDDYBEARS.LOCAL
 security = ads
 password server = ad.teddybears.local
 winbind separator = /
 winbind enum users = yes
 winbind enum groups = yes

Shortly after joining a domain though, winbindd will suddenly start using
100% CPU and sit there.

Attaching strace to this process shows nothing -- literally, it's making
no system calls whatsoever.  Attaching gdb to this running process and
using "next" reveals that it's doing the following:

memcpy() -> tdb_read() -> tdb_find()

... and then stops in tdb_find() claiming there's no more line numbers to
look at.  So on the tentative premise the problem lies in tbd_find(), I
didn't find anything obvious, but noticed the following call-chain is
happening:

tdb_find()
  |---> tdb_key_compare()
  +---> tdb_parse_data()

Not knowing *anything* about samba internals, can someone suggest whether
this is even a relevant starting point?

We're currently using samba 3.2.8 -- and in trying to diagnose this, have
gone through some releases, trying samba 3.2.10 and then samba 3.3.3 --
alas, this problem still remains in both those versions.  Our use of Samba
is deployed as part of a commercial product.  The kernel we're running is 
2.6.16.60-3-smp.

Is there something that we're not adding to our configuration file (as in
the above) which might aid in solving this?  It's certainly a difficult
problem to replicate -- and perhaps the only likely aspect worth
mentioning is that samba is being asked to join a domain against Windows
Server 2008 -- should that be relevant.

Having done a lot of background work trying to track this down, there's
almost no information about this in the mailing list archives which
suggests to me some very odd corner-case bug, a problem with Samba and
Windows Server 2008, or a configuration problem.  Given that the
configuration referenced above is the same for all customers (barring the
obvious changes for values on netbios, workgroup, realm, etc.) it's likely
to be something else, but I am open to suggestions.

I'm attaching a version of a backtrace from GDB of a running version of
winbindd exhibiting this problem.  The version of winbind is from version 
3.2.4.  There's two backtraces of the same process for comparison.

If there's any further information, don't hesitiate to let me know.

Kindly,

-- 
Thomas Adam
Senior Developer
Smoothwall Ltd.

Email:  thomas.adam at smoothwall.net

SmoothWall Limited
1 John Charles Way
Leeds LS12 6QA
United Kingdom

Phone: 1 800 959 3760     (USA, Canada and North America)
       0870 1 999 500     (United Kingdom)
       +44 870 1 999 500  (all other countries)
Fax:   +44 870 1 991 399

Web: http://www.smoothwall.net/

SmoothWall Limited is registered in England, Company Number: 429824i7

This email and any attachments transmitted with it are confidential to
the intended recipient(s) and may not be communicated to any other
person or published by any means without the express permission of
SmoothWall Limited.  Any views expressed in this message are solely
those of the author.  See:  http://smoothwall.net/company/email.php 
for the full text of this notice.
-------------- next part --------------
#0  0x402aa970 in tdb_find () from /modules/guardian/usr/lib/libtdb.so.1
#1  0x402aaa8d in tdb_update_hash () from /modules/guardian/usr/lib/libtdb.so.1
#2  0x402ab3c4 in tdb_store () from /modules/guardian/usr/lib/libtdb.so.1
#3  0x0812a48e in tdb_store_bystring ()
#4  0x0837f493 in netsamlogon_cache_store ()
#5  0x080aea38 in winbindd_dual_pam_auth_crap ()
#6  0x080c45d4 in child_process_request ()
#7  0x080c6a58 in fork_domain_child ()
#8  0x080c4191 in schedule_async_request ()
#9  0x080c3afe in async_request ()
#10 0x0809d467 in init_child_connection ()
#11 0x080c430e in async_domain_request ()
#12 0x0809cab8 in add_trusted_domains ()
#13 0x0809d1b8 in rescan_trusted_domains ()
#14 0x080946eb in process_loop ()
#15 0x0809571e in main ()


#0  0x4038979c in memcpy () from /lib/libc.so.6
#1  0x402af1a2 in tdb_read () from /modules/guardian/usr/lib/libtdb.so.1
#2  0x402afb6a in tdb_rec_read () from /modules/guardian/usr/lib/libtdb.so.1
#3  0x402aa90c in tdb_find () from /modules/guardian/usr/lib/libtdb.so.1
#4  0x402aaa8d in tdb_update_hash () from /modules/guardian/usr/lib/libtdb.so.1
#5  0x402ab3c4 in tdb_store () from /modules/guardian/usr/lib/libtdb.so.1
#6  0x0812a48e in tdb_store_bystring ()
#7  0x0837f493 in netsamlogon_cache_store ()
#8  0x080aea38 in winbindd_dual_pam_auth_crap ()
#9  0x080c45d4 in child_process_request ()
#10 0x080c6a58 in fork_domain_child ()
#11 0x080c4191 in schedule_async_request ()
#12 0x080c3afe in async_request ()
#13 0x0809d467 in init_child_connection ()
#14 0x080c430e in async_domain_request ()
#15 0x0809cab8 in add_trusted_domains ()
#16 0x0809d1b8 in rescan_trusted_domains ()
#17 0x080946eb in process_loop ()
#18 0x0809571e in main ()