[Samba] smbd processes 99% cpu and end-user locks up (good gdb attached)

Nathan Vidican nvidican at wmptl.com
Wed Oct 19 17:44:06 GMT 2005


Attached two more gdb's, seems to be the same output - working on compiling new 
openldap librairies now just thought to include in case something differed or 
sparked other ideas.


Gerald (Jerry) Carter wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Nathan Vidican wrote:
> 
> | Thanks, makes sense... so I waited for another (locked up)
> | pid, and did  a `gdb /usr/samba/sbin/smbd <pid>` , and
> | got a (hopefully useful) gdb output. I have atatched
> | the output here, along with the logfile of the
> | user which the smbd process belonged to when it happened.
> 
> Looks like the OpenLDAP client libs are stuck in a select()
> call.  What version of OL are you using?  Is there a newer version you
> could try?
> 
> 
> 
> (gdb) bt
> #0  0x0000000801cdc1dc in select () from /lib/libc.so.6
> #1  0x000000080097ab45 in ldap_result () from
> /usr/local/lib/libldap-2.2.so.7
> #2  0x000000080097f32b in ldap_sasl_bind_s () from
> /usr/local/lib/libldap-2.2.so.7
> #3  0x000000080097f830 in ldap_simple_bind_s () from
> /usr/local/lib/libldap-2.2.so.7
> #4  0x00000000006bc278 in smbldap_connect_system (ldap_state=0x90eb48,
> ldap_struct=0x991200) at lib/smbldap.c:780
> #5  0x00000000006bc787 in smbldap_open (ldap_state=0x90eb48) at
> lib/smbldap.c:860
> 
> 
> 
> 
> 
> 
> cheers, jerry
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.0 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
> 
> iD8DBQFDVmxCIR7qMdg1EfYRAsfeAJ9qzQGJoJ/v9mHJNRP8JPa5jr5iowCfW7wF
> N7SfAtnlJBizThAQP87QpDw=
> =qBu1
> -----END PGP SIGNATURE-----
> 
> 


-- 
Nathan Vidican
nvidican at wmptl.com
Windsor Match Plate & Tool Ltd.
http://www.wmptl.com/
-------------- next part --------------
wmptwo# top
last pid: 64237;  load averages:  1.18,  0.68,  0.30                                            up 3+20:40:56  13:36:02
72 processes:  3 running, 69 sleeping
CPU states: 10.3% user,  0.0% nice, 40.0% system,  0.0% interrupt, 49.7% idle
Mem: 59M Active, 1486M Inact, 226M Wired, 73M Cache, 213M Buf, 37M Free
Swap: 4087M Total, 48K Used, 4087M Free

  PID USERNAME       THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
63860 root             1 123    0 31072K  9000K CPU0   0   3:04 99.02% smbd
64222 asmyrnis         1  96    0 30680K  8684K select 1   0:00  0.05% smbd
  494 ldap             4  20    0   118M  7780K kserel 1   0:23  0.00% slapd
 7249 root             1  96    0 21460K  4184K select 0   0:15  0.00% cupsd
39470 rtalbot          1  96    0 30904K  8796K select 1   0:05  0.00% smbd
39505 root             1  96    0 31252K  9216K select 0   0:05  0.00% smbd
wmptwo#

wmptwo# /usr/samba/bin/net status sessions | grep asmyrnis
63860   asmyrnis      estimating    wmpest01     (10.0.0.22)
64222   asmyrnis      estimating    wmpest01     (10.0.0.22)
wmptwo#

wmptwo# gdb /usr/samba/sbin/smbd 63860
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Attaching to program: /usr/samba/sbin/smbd, process 63860
Reading symbols from /usr/local/lib/libldap-2.2.so.7...done.
Loaded symbols for /usr/local/lib/libldap-2.2.so.7
Reading symbols from /usr/local/lib/liblber-2.2.so.7...done.
Loaded symbols for /usr/local/lib/liblber-2.2.so.7
Reading symbols from /usr/lib/libgssapi.so.8...done.
Loaded symbols for /usr/lib/libgssapi.so.8
Reading symbols from /usr/lib/libkrb5.so.8...done.
Loaded symbols for /usr/lib/libkrb5.so.8
Reading symbols from /usr/lib/libasn1.so.8...done.
Loaded symbols for /usr/lib/libasn1.so.8
Reading symbols from /lib/libcrypto.so.4...done.
Loaded symbols for /lib/libcrypto.so.4
Reading symbols from /usr/lib/libroken.so.8...done.
Loaded symbols for /usr/lib/libroken.so.8
Reading symbols from /lib/libcrypt.so.3...done.
Loaded symbols for /lib/libcrypt.so.3
Reading symbols from /usr/lib/libcom_err.so.3...done.
Loaded symbols for /usr/lib/libcom_err.so.3
Reading symbols from /usr/local/lib/libcups.so.2...done.
Loaded symbols for /usr/local/lib/libcups.so.2
Reading symbols from /usr/local/lib/libgnutls.so.12...done.
Loaded symbols for /usr/local/lib/libgnutls.so.12
Reading symbols from /usr/local/lib/libgcrypt.so.13...done.
Loaded symbols for /usr/local/lib/libgcrypt.so.13
Reading symbols from /usr/local/lib/libgpg-error.so.1...done.
Loaded symbols for /usr/local/lib/libgpg-error.so.1
Reading symbols from /lib/libz.so.3...done.
Loaded symbols for /lib/libz.so.3
Reading symbols from /usr/local/lib/libiconv.so.3...done.
Loaded symbols for /usr/local/lib/libiconv.so.3
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /usr/lib/libssl.so.4...done.
Loaded symbols for /usr/lib/libssl.so.4
Reading symbols from /usr/local/lib/libintl.so.6...done.
Loaded symbols for /usr/local/lib/libintl.so.6
Reading symbols from /usr/local/lib/nss_ldap.so.1...done.
Loaded symbols for /usr/local/lib/nss_ldap.so.1
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
0x0000000801cdc1dc in select () from /lib/libc.so.6
(gdb)
#0  0x0000000801cdc1dc in select () from /lib/libc.so.6
#1  0x000000080097ab45 in ldap_result () from /usr/local/lib/libldap-2.2.so.7
#2  0x000000080097f32b in ldap_sasl_bind_s () from /usr/local/lib/libldap-2.2.so.7
#3  0x000000080097f830 in ldap_simple_bind_s () from /usr/local/lib/libldap-2.2.so.7
#4  0x00000000006bc278 in smbldap_connect_system (ldap_state=0x90ee48, ldap_struct=0x9b0600) at lib/smbldap.c:780
#5  0x00000000006bc787 in smbldap_open (ldap_state=0x90ee48) at lib/smbldap.c:860
#6  0x00000000006bca2b in another_ldap_try (ldap_state=0x90ee48, rc=0x7fffffffd4d0, attempts=0x7fffffffd4cc,
    endtime=1129743194) at lib/smbldap.c:937
#7  0x00000000006bce7f in smbldap_search_ext (ldap_state=0x90ee48, base=0x9891c8 "dc=wmptl,dc=net", scope=2,
    filter=0x7fffffffd6f0 "(&(sambaSID=S-1-5-21-2818898263-1345796712-1011627658-5017)(objectclass=sambaSamAccount))",
    attrs=0x950600, attrsonly=0, sctrls=0x0, cctrls=0x0, sizelimit=0, res=0x7fffffffdcb8) at lib/smbldap.c:1026
#8  0x00000000006bcfad in smbldap_search (ldap_state=0x90ee48, base=0x9891c8 "dc=wmptl,dc=net", scope=2,
    filter=0x7fffffffd6f0 "(&(sambaSID=S-1-5-21-2818898263-1345796712-1011627658-5017)(objectclass=sambaSamAccount))",
    attrs=0x950600, attrsonly=0, res=0x7fffffffdcb8) at lib/smbldap.c:1050
#9  0x00000000006bdaf1 in smbldap_search_suffix (ldap_state=0x90ee48,
    filter=0x7fffffffd6f0 "(&(sambaSID=S-1-5-21-2818898263-1345796712-1011627658-5017)(objectclass=sambaSamAccount))",
    search_attr=0x950600, result=0x7fffffffdcb8) at lib/smbldap.c:1239
#10 0x00000000005eadfd in ldapsam_search_suffix_by_sid (ldap_state=0x90ed48, sid=0x8aa698, result=0x7fffffffdcb8,
    attr=0x950600) at passdb/pdb_ldap.c:252
#11 0x00000000005ee6b8 in ldapsam_get_ldap_user_by_sid (ldap_state=0x90ed48, sid=0x8aa698, result=0x7fffffffdcb8)
    at passdb/pdb_ldap.c:1372
#12 0x00000000005ee787 in ldapsam_getsampwsid (my_methods=0x920048, user=0x9a5448, sid=0x8aa698)
    at passdb/pdb_ldap.c:1408
#13 0x00000000005e21a1 in context_getsampwsid (context=0x8d0a48, sam_acct=0x9a5448, sid=0x8aa698)
    at passdb/pdb_interface.c:222
#14 0x00000000005e4b7f in pdb_getsampwsid (sam_acct=0x9a5448, sid=0x8aa698) at passdb/pdb_interface.c:1015
#15 0x00000000005dddbc in local_sid_to_uid (puid=0x8aa6e4, psid=0x8aa698, name_type=0x7fffffffdddc)
    at passdb/passdb.c:1193
#16 0x00000000005e98c9 in sid_to_uid (psid=0x8aa698, puid=0x8aa6e4) at passdb/lookup_sid.c:395
#17 0x00000000004b9863 in create_canon_ace_lists (fsp=0x9b0100, pst=0x7fffffffe540, pfile_owner_sid=0x7fffffffe4f0,
    pfile_grp_sid=0x7fffffffe4a0, ppfile_ace=0x7fffffffe3d0, ppdir_ace=0x7fffffffe3c8, dacl=0x921c48)
    at smbd/posix_acls.c:1393
#18 0x00000000004ba92e in unpack_canon_ace (fsp=0x9b0100, pst=0x7fffffffe540, pfile_owner_sid=0x7fffffffe4f0,
    pfile_grp_sid=0x7fffffffe4a0, ppfile_ace=0x7fffffffe498, ppdir_ace=0x7fffffffe490, security_info_sent=7,
    psd=0x921bc8) at smbd/posix_acls.c:1945
#19 0x00000000004bebd7 in set_nt_acl (fsp=0x9b0100, security_info_sent=7, psd=0x921bc8) at smbd/posix_acls.c:3175
#20 0x00000000004b48da in vfswrap_fset_nt_acl (handle=0x0, fsp=0x9b0100, fd=-1, security_info_sent=7, psd=0x921bc8)
    at smbd/vfs-wrap.c:837
#21 0x000000000046ec52 in set_sd (fsp=0x9b0100, data=0x9b0200 "\001", sd_len=176, security_info_sent=7)
    at smbd/nttrans.c:1041
#22 0x000000000047283c in call_nt_transact_set_security_desc (conn=0x995048, inbuf=0x92f000 "", outbuf=0x961000 "",
---Type <return> to continue, or q <return> to quit---
    length=264, bufsize=131072, ppsetup=0x7fffffffe7c0, setup_count=0, ppparams=0x7fffffffe7d0, parameter_count=8,
    ppdata=0x7fffffffe7c8, data_count=176, max_data_count=0) at smbd/nttrans.c:2087
#23 0x0000000000474aca in reply_nttrans (conn=0x995048, inbuf=0x92f000 "", outbuf=0x961000 "", length=264,
    bufsize=131072) at smbd/nttrans.c:2990
#24 0x00000000004c4792 in switch_message (type=160, inbuf=0x92f000 "", outbuf=0x961000 "", size=264, bufsize=131072)
    at smbd/process.c:991
#25 0x00000000004c4866 in construct_reply (inbuf=0x92f000 "", outbuf=0x961000 "", size=264, bufsize=131072)
    at smbd/process.c:1021
#26 0x00000000004c4c3e in process_smb (inbuf=0x92f000 "", outbuf=0x961000 "") at smbd/process.c:1121
#27 0x00000000004c5ecf in smbd_process () at smbd/process.c:1670
#28 0x00000000006c12e2 in main (argc=2, argv=0x7fffffffebb8) at smbd/server.c:975
(gdb) quit
wmptwo#
-------------- next part --------------
wmptwo#
last pid: 64203;  load averages:  0.44,  0.11,  0.04                                            up 3+20:37:22  13:32:28
70 processes:  1 running, 69 sleeping
CPU states: 14.0% user,  0.0% nice, 36.6% system,  0.0% interrupt, 49.5% idle
Mem: 59M Active, 1486M Inact, 226M Wired, 73M Cache, 213M Buf, 38M Free
Swap: 4087M Total, 48K Used, 4087M Free

  PID USERNAME       THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
63905 root             1 110    0 30836K  8704K CPU0   0   0:32 78.76% smbd
  494 ldap             4  20    0   118M  7784K kserel 1   0:22  0.00% slapd
 7249 root             1  96    0 21460K  4184K select 0   0:15  0.00% cupsd
39470 rtalbot          1  96    0 30904K  8796K select 1   0:05  0.00% smbd
39505 pcarlini         1  96    0 31252K  9260K select 1   0:04  0.00% smbd
wmptwo#


wmptwo# /usr/samba/bin/net status sessions | grep ccope
63905   ccope         humanresources  wmpoff28     (10.0.0.17)
64230   ccope         humanresources  wmpoff28     (10.0.0.17)
wmptwo#


wmptwo# gdb /usr/samba/sbin/smbd 63905
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Attaching to program: /usr/samba/sbin/smbd, process 63905
Reading symbols from /usr/local/lib/libldap-2.2.so.7...done.
Loaded symbols for /usr/local/lib/libldap-2.2.so.7
Reading symbols from /usr/local/lib/liblber-2.2.so.7...done.
Loaded symbols for /usr/local/lib/liblber-2.2.so.7
Reading symbols from /usr/lib/libgssapi.so.8...done.
Loaded symbols for /usr/lib/libgssapi.so.8
Reading symbols from /usr/lib/libkrb5.so.8...done.
Loaded symbols for /usr/lib/libkrb5.so.8
Reading symbols from /usr/lib/libasn1.so.8...done.
Loaded symbols for /usr/lib/libasn1.so.8
Reading symbols from /lib/libcrypto.so.4...done.
Loaded symbols for /lib/libcrypto.so.4
Reading symbols from /usr/lib/libroken.so.8...done.
Loaded symbols for /usr/lib/libroken.so.8
Reading symbols from /lib/libcrypt.so.3...done.
Loaded symbols for /lib/libcrypt.so.3
Reading symbols from /usr/lib/libcom_err.so.3...done.
Loaded symbols for /usr/lib/libcom_err.so.3
Reading symbols from /usr/local/lib/libcups.so.2...done.
Loaded symbols for /usr/local/lib/libcups.so.2
Reading symbols from /usr/local/lib/libgnutls.so.12...done.
Loaded symbols for /usr/local/lib/libgnutls.so.12
Reading symbols from /usr/local/lib/libgcrypt.so.13...done.
Loaded symbols for /usr/local/lib/libgcrypt.so.13
Reading symbols from /usr/local/lib/libgpg-error.so.1...done.
Loaded symbols for /usr/local/lib/libgpg-error.so.1
Reading symbols from /lib/libz.so.3...done.
Loaded symbols for /lib/libz.so.3
Reading symbols from /usr/local/lib/libiconv.so.3...done.
Loaded symbols for /usr/local/lib/libiconv.so.3
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /usr/lib/libssl.so.4...done.
Loaded symbols for /usr/lib/libssl.so.4
Reading symbols from /usr/local/lib/libintl.so.6...done.
Loaded symbols for /usr/local/lib/libintl.so.6
Reading symbols from /usr/local/lib/nss_ldap.so.1...done.
Loaded symbols for /usr/local/lib/nss_ldap.so.1
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
0x0000000801cdc1dc in select () from /lib/libc.so.6
(gdb)
(gdb) bt
#0  0x0000000801cdc1dc in select () from /lib/libc.so.6
#1  0x000000080097ab45 in ldap_result () from /usr/local/lib/libldap-2.2.so.7
#2  0x000000080097f32b in ldap_sasl_bind_s () from /usr/local/lib/libldap-2.2.so.7
#3  0x000000080097f830 in ldap_simple_bind_s () from /usr/local/lib/libldap-2.2.so.7
#4  0x00000000006bc278 in smbldap_connect_system (ldap_state=0x90ee48, ldap_struct=0x959000) at lib/smbldap.c:780
#5  0x00000000006bc787 in smbldap_open (ldap_state=0x90ee48) at lib/smbldap.c:860
#6  0x00000000006bca2b in another_ldap_try (ldap_state=0x90ee48, rc=0x7fffffffd4d0, attempts=0x7fffffffd4cc,
    endtime=1129743131) at lib/smbldap.c:937
#7  0x00000000006bce7f in smbldap_search_ext (ldap_state=0x90ee48, base=0x8aa448 "dc=wmptl,dc=net", scope=2,
    filter=0x7fffffffd6f0 "(&(sambaSID=S-1-5-21-2818898263-1345796712-1011627658-5011)(objectclass=sambaSamAccount))",
    attrs=0x8d0e00, attrsonly=0, sctrls=0x0, cctrls=0x0, sizelimit=0, res=0x7fffffffdcb8) at lib/smbldap.c:1026
#8  0x00000000006bcfad in smbldap_search (ldap_state=0x90ee48, base=0x8aa448 "dc=wmptl,dc=net", scope=2,
    filter=0x7fffffffd6f0 "(&(sambaSID=S-1-5-21-2818898263-1345796712-1011627658-5011)(objectclass=sambaSamAccount))",
    attrs=0x8d0e00, attrsonly=0, res=0x7fffffffdcb8) at lib/smbldap.c:1050
#9  0x00000000006bdaf1 in smbldap_search_suffix (ldap_state=0x90ee48,
    filter=0x7fffffffd6f0 "(&(sambaSID=S-1-5-21-2818898263-1345796712-1011627658-5011)(objectclass=sambaSamAccount))",
    search_attr=0x8d0e00, result=0x7fffffffdcb8) at lib/smbldap.c:1239
#10 0x00000000005eadfd in ldapsam_search_suffix_by_sid (ldap_state=0x90ed48, sid=0x8a2e98, result=0x7fffffffdcb8,
    attr=0x8d0e00) at passdb/pdb_ldap.c:252
#11 0x00000000005ee6b8 in ldapsam_get_ldap_user_by_sid (ldap_state=0x90ed48, sid=0x8a2e98, result=0x7fffffffdcb8)
    at passdb/pdb_ldap.c:1372
#12 0x00000000005ee787 in ldapsam_getsampwsid (my_methods=0x920048, user=0x956448, sid=0x8a2e98)
    at passdb/pdb_ldap.c:1408
#13 0x00000000005e21a1 in context_getsampwsid (context=0x8d0a48, sam_acct=0x956448, sid=0x8a2e98)
    at passdb/pdb_interface.c:222
#14 0x00000000005e4b7f in pdb_getsampwsid (sam_acct=0x956448, sid=0x8a2e98) at passdb/pdb_interface.c:1015
#15 0x00000000005dddbc in local_sid_to_uid (puid=0x8a2ee4, psid=0x8a2e98, name_type=0x7fffffffdddc)
    at passdb/passdb.c:1193
#16 0x00000000005e98c9 in sid_to_uid (psid=0x8a2e98, puid=0x8a2ee4) at passdb/lookup_sid.c:395
#17 0x00000000004b9863 in create_canon_ace_lists (fsp=0x922500, pst=0x7fffffffe540, pfile_owner_sid=0x7fffffffe4f0,
    pfile_grp_sid=0x7fffffffe4a0, ppfile_ace=0x7fffffffe3d0, ppdir_ace=0x7fffffffe3c8, dacl=0x8a2648)
    at smbd/posix_acls.c:1393
#18 0x00000000004ba92e in unpack_canon_ace (fsp=0x922500, pst=0x7fffffffe540, pfile_owner_sid=0x7fffffffe4f0,
    pfile_grp_sid=0x7fffffffe4a0, ppfile_ace=0x7fffffffe498, ppdir_ace=0x7fffffffe490, security_info_sent=2147483652,
    psd=0x8a25c8) at smbd/posix_acls.c:1945
#19 0x00000000004bebd7 in set_nt_acl (fsp=0x922500, security_info_sent=2147483652, psd=0x8a25c8)
    at smbd/posix_acls.c:3175
#20 0x00000000004b48da in vfswrap_fset_nt_acl (handle=0x0, fsp=0x922500, fd=-1, security_info_sent=2147483652,
    psd=0x8a25c8) at smbd/vfs-wrap.c:837
#21 0x000000000046ec52 in set_sd (fsp=0x922500, data=0x922600 "\001", sd_len=164, security_info_sent=2147483652)
    at smbd/nttrans.c:1041
---Type <return> to continue, or q <return> to quit---
#22 0x000000000047283c in call_nt_transact_set_security_desc (conn=0x955048, inbuf=0x92f000 "", outbuf=0x961000 "",
    length=252, bufsize=131072, ppsetup=0x7fffffffe7c0, setup_count=0, ppparams=0x7fffffffe7d0, parameter_count=8,
    ppdata=0x7fffffffe7c8, data_count=164, max_data_count=0) at smbd/nttrans.c:2087
#23 0x0000000000474aca in reply_nttrans (conn=0x955048, inbuf=0x92f000 "", outbuf=0x961000 "", length=252,
    bufsize=131072) at smbd/nttrans.c:2990
#24 0x00000000004c4792 in switch_message (type=160, inbuf=0x92f000 "", outbuf=0x961000 "", size=252, bufsize=131072)
    at smbd/process.c:991
#25 0x00000000004c4866 in construct_reply (inbuf=0x92f000 "", outbuf=0x961000 "", size=252, bufsize=131072)
    at smbd/process.c:1021
#26 0x00000000004c4c3e in process_smb (inbuf=0x92f000 "", outbuf=0x961000 "") at smbd/process.c:1121
#27 0x00000000004c5ecf in smbd_process () at smbd/process.c:1670
#28 0x00000000006c12e2 in main (argc=2, argv=0x7fffffffebb8) at smbd/server.c:975
(gdb)
wmptwo#


More information about the samba mailing list