talloc-2.1.12 issues with library destructor
Lukas Slebodnik
lslebodn at redhat.com
Mon Mar 26 09:49:05 UTC 2018
On (24/03/18 22:51), Lukas Slebodnik via samba-technical wrote:
>ehlo,
>
>The latest version of libtdb changed handling of releaseing autofree_context.
>Previously, it was release with atexit and currently tith library destructor.
>
>However, it caused some crashes in sssd test suite when shuttind down
>processes. In one case, there was an abort in krb5 library.
>
> Stack trace of thread 19667:
> #0 0x00007f2cab91ff6b __GI_raise (libc.so.6)
> #1 0x00007f2cab90a5c1 __GI_abort (libc.so.6)
> #2 0x00007f2cab90a491 __assert_fail_base (libc.so.6)
> #3 0x00007f2cab9186e2 __GI___assert_fail (libc.so.6)
> #4 0x00007f2cb10aaca5 k5_mutex_lock (libkrb5.so.3)
> #5 0x00007f2cb10ab790 k5_mutex_lock (libkrb5.so.3)
> #6 0x00007f2cb10ab8f5 profile_free_file (libkrb5.so.3)
> #7 0x00007f2cb10ab983 profile_close_file (libkrb5.so.3)
> #8 0x00007f2cb10af249 profile_release (libkrb5.so.3)
> #9 0x00007f2cb10a06c7 k5_os_free_context (libkrb5.so.3)
> #10 0x00007f2cb1075a9a krb5_free_context (libkrb5.so.3)
> #11 0x000055cea7cb2dd1 kcm_data_destructor (sssd_kcm)
> #12 0x00007f2cac153e96 _tc_free_internal (libtalloc.so.2)
> #13 0x00007f2cac1537b0 _tc_free_internal (libtalloc.so.2)
> #14 0x00007f2cac1537b0 _tc_free_internal (libtalloc.so.2)
> #15 0x00007f2cac1537b0 _tc_free_internal (libtalloc.so.2)
> #16 0x00007f2cac1537b0 _tc_free_internal (libtalloc.so.2)
> #17 0x00007f2cac14e648 _talloc_free (libtalloc.so.2)
> #18 0x00007f2cac14c480 talloc_lib_fini (libtalloc.so.2)
> #19 0x00007f2cb151da96 _dl_fini (ld-linux-x86-64.so.2)
> #20 0x00007f2cab9226bc __run_exit_handlers (libc.so.6)
> #21 0x00007f2cab9227ec __GI_exit (libc.so.6)
> #22 0x00007f2cb030dc61 orderly_shutdown (libsss_util.so)
> #23 0x00007f2cac365a46 tevent_common_check_signal (libtevent.so.0)
> #24 0x00007f2cac367975 epoll_event_loop_once (libtevent.so.0)
> #25 0x00007f2cac365dab std_event_loop_once (libtevent.so.0)
> #26 0x00007f2cac362098 _tevent_loop_once (libtevent.so.0)
> #27 0x00007f2cac3622eb tevent_common_loop_wait (libtevent.so.0)
> #28 0x00007f2cac365d3b std_event_loop_wait (libtevent.so.0)
> #29 0x00007f2cb030eb37 server_loop (libsss_util.so)
> #30 0x000055cea7cb29f4 main (sssd_kcm)
> #31 0x00007f2cab90c1eb __libc_start_main (libc.so.6)
> #32 0x000055cea7cb2c7a _start (sssd_kcm)
>
>Previously, it was not problem because atexit/on_exit are executed before
>destructors. Nice description is in a blog[1].
>
>Another crash is in nss_wrapper(in attachment) because gethostname was called
>by libldap which tried to initialize some structures (ldap_int_initialize).
>And it is really unexpected that libldap tries to initialize some structures
>at exit. I would say it tries to initialize something which was already
>released otherwise ldap_int_initialize would not be called
>
And of course, I forgot to attach the file :-)
BTW it crashed in nss_wrapper due to dereference of null pointer
in macro nwrap_load_lib_function.
Explanation is quite simple: nwrap_destructor was already executed
and nobody expected that nss_wrapper will be used after calling destructor.
LS
>Maybe it could be solved with priority of destructors but documentation
>does not say anything about default priority.
>
>I was also not able to find anything about order of destructors between
>libraries.
>
>On the one hand it happens just at shutdown but on the other hand it is
>still a crash. Can you see some crashes even with samba? I am not sure
>whether you have configured some way to detect crashes which does not cause
>failures in autobuild
>(systemd-coredumpd ...)
>
>LS
>
>[1] http://ptspts.blogspot.cz/2014/01/how-to-run-custom-code-before-and-after.html
>[2] https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-destructor-function-attribute
>
-------------- next part --------------
GNU gdb (GDB) Fedora 8.1-11.fc29
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /tmp/sssd-intg.mqhouxnn/libexec/sssd/sssd_be...done.
[New LWP 2789]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/tmp/sssd-intg.mqhouxnn/libexec/sssd/sssd_be --domain LDAP --uid 0 --gid 0 --lo'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fb45db13bdb in libc_gethostname (len=64, name=0x7ffe7d231b70 "\260\062\060\066ae00") at /usr/src/debug/nss_wrapper-1.1.3-3.fc27.x86_64/src/nss_wrapper.c:1285
1285 nwrap_load_lib_function(NWRAP_LIBNSL, gethostname);
#0 0x00007fb45db13bdb in libc_gethostname (len=64, name=0x7ffe7d231b70 "\260\062\060\066ae00") at /usr/src/debug/nss_wrapper-1.1.3-3.fc27.x86_64/src/nss_wrapper.c:1285
No locals.
#1 gethostname (name=0x7ffe7d231b70 "\260\062\060\066ae00", len=64) at /usr/src/debug/nss_wrapper-1.1.3-3.fc27.x86_64/src/nss_wrapper.c:5526
No locals.
#2 0x00007fb45bb7d0e9 in gethostname (__buflen=64, __buf=0x7ffe7d231b70 "\260\062\060\066ae00") at /usr/include/bits/unistd.h:354
No locals.
#3 ldap_pvt_get_fqdn (name=name at entry=0x0) at util-int.c:843
fqdn = <optimized out>
ha_buf = 0x2000 <error: Cannot access memory at address 0x2000>
hostbuf = "\260\062\060\066ae00\000\031\023\212\312\067\066\060\v\000\000\000\000\000\000\000p\236\003\002\000\000\000\000\240b\244Z\264\177\000\000-\002pZ\264\177\000\000\342\a\000\000\000\000\000\000\060\221\345Z\264\177\000\000"
hp = 0x7e2
he_buf = {h_name = 0x7ffe7d231bc0 "3\275\270Z", h_aliases = 0x7fb45a70c959 <_IO_new_do_write+25>, h_addrtype = 33791600, h_length = 0, h_addr_list = 0x7fb45a70aa28 <_IO_new_file_sync+184>}
rc = <optimized out>
local_h_errno = 32692
#4 0x00007fb45bb7b2de in ldap_int_initialize (gopts=gopts at entry=0x7fb45bd9c040 <ldap_int_global_options>, dbglvl=dbglvl at entry=0x0) at init.c:648
name = 0x0
#5 0x00007fb45bb7ba8a in ldap_get_option (ld=0x206ae00, option=option at entry=20497, outvalue=outvalue at entry=0x2068ee0) at options.c:108
lo = <optimized out>
rc = -1
__PRETTY_FUNCTION__ = "ldap_get_option"
#6 0x00007fb4469a6c13 in remove_connection_callback (mem_ctx=mem_ctx at entry=0x2068ee0) at src/providers/ldap/sdap_fd_events.c:75
lret = <optimized out>
conncb = 0x2068ee0
cb_data = <optimized out>
__FUNCTION__ = "remove_connection_callback"
__debug_macro_level = <optimized out>
__debug_macro_level = <optimized out>
#7 0x00007fb45d2e2d2b in _tc_free_internal (location=0x7fb4469e2260 "src/providers/ldap/sdap_fd_events.c:57", tc=0x2068e80) at ../talloc.c:1145
d = 0x7fb4469a6be0 <remove_connection_callback>
ptr_to_free = <optimized out>
ptr = 0x2068ee0
ptr_to_free = <optimized out>
ptr = <optimized out>
is_child = <optimized out>
d = <optimized out>
pool = <optimized out>
_flen = <optimized out>
_fptr = <optimized out>
#8 _talloc_free_internal (location=0x7fb4469e2260 "src/providers/ldap/sdap_fd_events.c:57", ptr=0x2068ee0) at ../talloc.c:1235
tc = 0x2068e80
tc = <optimized out>
fill = <optimized out>
#9 _talloc_free () at ../talloc.c:1777
tc = 0x2068e80
#10 0x00007fb4469a6d2c in remove_ldap_connection_callbacks (sh=sh at entry=0x205a600) at src/providers/ldap/sdap_fd_events.c:57
No locals.
#11 0x00007fb44697ff82 in sdap_handle_release (sh=0x205a600) at src/providers/ldap/sdap_async.c:99
op = <optimized out>
op = <optimized out>
__FUNCTION__ = "sdap_handle_release"
__debug_macro_level = <optimized out>
_talloc_destructor_fn = <optimized out>
#12 sdap_handle_destructor (mem=mem at entry=0x205a600) at src/providers/ldap/sdap_async.c:79
sh = 0x205a600
#13 0x00007fb45d2e7e96 in _tc_free_internal () at ../talloc.c:1145
d = 0x7fb44697ff40 <sdap_handle_destructor>
ptr_to_free = <optimized out>
ptr = 0x205a600
is_child = <optimized out>
d = <optimized out>
pool = <optimized out>
_flen = <optimized out>
_fptr = <optimized out>
#14 0x00007fb45d2e77b0 in _tc_free_children_internal (location=0x7fb45d2ed519 "../talloc.c:440", ptr=0x205a430, tc=0x205a3d0) at ../talloc.c:1654
child = 0x205a600
new_parent = 0x0
#15 _tc_free_internal () at ../talloc.c:1171
ptr_to_free = <optimized out>
ptr = 0x205a430
is_child = <optimized out>
d = <optimized out>
pool = <optimized out>
_flen = <optimized out>
_fptr = <optimized out>
#16 0x00007fb45d2e77b0 in _tc_free_children_internal (location=0x7fb45d2ed519 "../talloc.c:440", ptr=0x205ce10, tc=0x205cdb0) at ../talloc.c:1654
child = 0x205a430
new_parent = 0x0
#17 _tc_free_internal () at ../talloc.c:1171
ptr_to_free = <optimized out>
ptr = 0x205ce10
is_child = <optimized out>
d = <optimized out>
pool = <optimized out>
_flen = <optimized out>
_fptr = <optimized out>
#18 0x00007fb45d2e77b0 in _tc_free_children_internal (location=0x7fb45d2ed519 "../talloc.c:440", ptr=0x205e3e0, tc=0x205e380) at ../talloc.c:1654
child = 0x205ce10
new_parent = 0x0
#19 _tc_free_internal () at ../talloc.c:1171
ptr_to_free = <optimized out>
ptr = 0x205e3e0
is_child = <optimized out>
d = <optimized out>
pool = <optimized out>
_flen = <optimized out>
_fptr = <optimized out>
#20 0x00007fb45d2e77b0 in _tc_free_children_internal (location=0x7fb45d2ed519 "../talloc.c:440", ptr=0x205ec70, tc=0x205ec10) at ../talloc.c:1654
child = 0x205e3e0
new_parent = 0x0
#21 _tc_free_internal () at ../talloc.c:1171
ptr_to_free = <optimized out>
ptr = 0x205ec70
is_child = <optimized out>
d = <optimized out>
pool = <optimized out>
_flen = <optimized out>
_fptr = <optimized out>
#22 0x00007fb45d2e77b0 in _tc_free_children_internal (location=0x7fb45d2ed519 "../talloc.c:440", ptr=0x20550c0, tc=0x2055060) at ../talloc.c:1654
child = 0x205ec70
new_parent = 0x0
#23 _tc_free_internal () at ../talloc.c:1171
ptr_to_free = <optimized out>
ptr = 0x20550c0
is_child = <optimized out>
d = <optimized out>
pool = <optimized out>
_flen = <optimized out>
_fptr = <optimized out>
#24 0x00007fb45d2e77b0 in _tc_free_children_internal (location=0x7fb45d2ed519 "../talloc.c:440", ptr=0x2056e70, tc=0x2056e10) at ../talloc.c:1654
child = 0x20550c0
new_parent = 0x0
#25 _tc_free_internal () at ../talloc.c:1171
ptr_to_free = <optimized out>
ptr = 0x2056e70
is_child = <optimized out>
d = <optimized out>
pool = <optimized out>
_flen = <optimized out>
_fptr = <optimized out>
#26 0x00007fb45d2e77b0 in _tc_free_children_internal (location=0x7fb45d2ed519 "../talloc.c:440", ptr=0x20554e0, tc=0x2055480) at ../talloc.c:1654
child = 0x2056e70
new_parent = 0x0
#27 _tc_free_internal () at ../talloc.c:1171
ptr_to_free = <optimized out>
ptr = 0x20554e0
is_child = <optimized out>
d = <optimized out>
pool = <optimized out>
_flen = <optimized out>
_fptr = <optimized out>
#28 0x00007fb45d2e77b0 in _tc_free_children_internal (location=0x7fb45d2ed519 "../talloc.c:440", ptr=0x203a270, tc=0x203a210) at ../talloc.c:1654
child = 0x20554e0
new_parent = 0x0
#29 _tc_free_internal () at ../talloc.c:1171
ptr_to_free = <optimized out>
ptr = 0x203a270
is_child = <optimized out>
d = <optimized out>
pool = <optimized out>
_flen = <optimized out>
_fptr = <optimized out>
#30 0x00007fb45d2e77b0 in _tc_free_children_internal (location=0x7fb45d2ed519 "../talloc.c:440", ptr=0x2038080, tc=0x2038020) at ../talloc.c:1654
child = 0x203a270
new_parent = 0x0
#31 _tc_free_internal () at ../talloc.c:1171
ptr_to_free = <optimized out>
ptr = 0x2038080
is_child = <optimized out>
d = <optimized out>
pool = <optimized out>
_flen = <optimized out>
_fptr = <optimized out>
#32 0x00007fb45d2e77b0 in _tc_free_children_internal (location=0x7fb45d2ed519 "../talloc.c:440", ptr=0x2030110, tc=0x20300b0) at ../talloc.c:1654
child = 0x2038080
new_parent = 0x0
#33 _tc_free_internal () at ../talloc.c:1171
ptr_to_free = <optimized out>
ptr = 0x2030110
is_child = <optimized out>
d = <optimized out>
pool = <optimized out>
_flen = <optimized out>
_fptr = <optimized out>
#34 0x00007fb45d2e77b0 in _tc_free_children_internal (location=0x7fb45d2ed519 "../talloc.c:440", ptr=0x202ec80, tc=0x202ec20) at ../talloc.c:1654
child = 0x2030110
new_parent = 0x0
#35 _tc_free_internal () at ../talloc.c:1171
ptr_to_free = <optimized out>
ptr = 0x202ec80
is_child = <optimized out>
d = <optimized out>
pool = <optimized out>
_flen = <optimized out>
_fptr = <optimized out>
#36 0x00007fb45d2e2648 in _tc_free_children_internal (location=0x7fb45d2ed519 "../talloc.c:440", ptr=0x202ea60, tc=0x202ea00) at ../talloc.c:1654
child = 0x202ec80
new_parent = 0x0
#37 _tc_free_internal (location=0x7fb45d2ed519 "../talloc.c:440", tc=0x202ea00) at ../talloc.c:1171
ptr_to_free = <optimized out>
ptr = 0x202ea60
ptr_to_free = <optimized out>
ptr = <optimized out>
is_child = <optimized out>
d = <optimized out>
pool = <optimized out>
_flen = <optimized out>
_fptr = <optimized out>
#38 _talloc_free_internal (location=0x7fb45d2ed519 "../talloc.c:440", ptr=0x202ea60) at ../talloc.c:1235
tc = 0x202ea00
tc = <optimized out>
fill = <optimized out>
#39 _talloc_free () at ../talloc.c:1777
tc = 0x202ea00
#40 0x00007fb45d2e0480 in talloc_lib_fini () at ../talloc.c:440
No locals.
#41 0x00007fb45e13aa96 in _dl_fini () at dl-fini.c:138
do_audit = <optimized out>
__PRETTY_FUNCTION__ = "_dl_fini"
#42 0x00007fb45a6c96bc in __run_exit_handlers (status=status at entry=0, listp=0x7fb45aa49718 <__exit_funcs>, run_list_atexit=run_list_atexit at entry=true, run_dtors=run_dtors at entry=true) at exit.c:108
atfct = <optimized out>
onfct = <optimized out>
cxafct = <optimized out>
f = <optimized out>
new_exitfn_called = 2
cur = 0x7fb45aa4ad80 <initial>
#43 0x00007fb45a6c97ec in __GI_exit (status=status at entry=0) at exit.c:139
No locals.
#44 0x00007fb45b0b0d9d in orderly_shutdown (status=0) at src/util/server.c:258
sent_sigterm = 1
__FUNCTION__ = "orderly_shutdown"
#45 0x00007fb45d4f9a46 in tevent_common_check_signal (ev=<optimized out>) at ../tevent_signal.c:417
se = 0x202fe50
exists = 0x207b650
count = 1
sl = <optimized out>
next = 0x0
counter = {count = 1, seen = 0}
clear_processed_siginfo = false
i = 15
#46 0x00007fb45d4fb975 in epoll_event_loop (tvalp=0x7ffe7d232750, epoll_ev=0x202ef00) at ../tevent_epoll.c:647
ret = -1
i = <optimized out>
timeout = <optimized out>
wait_errno = 4
events = {{events = 17, data = {ptr = 0x20687a0, fd = 33982368, u32 = 33982368, u64 = 33982368}}}
ret = <optimized out>
i = <optimized out>
events = <optimized out>
timeout = <optimized out>
wait_errno = <optimized out>
fde = <optimized out>
flags = <optimized out>
mpx_fde = <optimized out>
handled_fde = <optimized out>
handled_mpx = <optimized out>
#47 epoll_event_loop_once (ev=<optimized out>, location=<optimized out>) at ../tevent_epoll.c:930
epoll_ev = 0x202ef00
tval = {tv_sec = 5, tv_usec = 678229}
panic_triggered = false
#48 0x00007fb45d4f9dab in std_event_loop_once (ev=0x202ec80, location=0x7fb45b0d8b50 "src/util/server.c:721") at ../tevent_standard.c:114
glue_ptr = <optimized out>
glue = 0x202edb0
ret = <optimized out>
#49 0x00007fb45d4f6098 in _tevent_loop_once (ev=ev at entry=0x202ec80, location=location at entry=0x7fb45b0d8b50 "src/util/server.c:721") at ../tevent.c:725
ret = <optimized out>
nesting_stack_ptr = 0x0
#50 0x00007fb45d4f62eb in tevent_common_loop_wait (ev=0x202ec80, location=0x7fb45b0d8b50 "src/util/server.c:721") at ../tevent.c:848
ret = <optimized out>
#51 0x00007fb45d4f9d3b in std_event_loop_wait (ev=0x202ec80, location=0x7fb45b0d8b50 "src/util/server.c:721") at ../tevent_standard.c:145
glue_ptr = <optimized out>
glue = 0x202edb0
ret = <optimized out>
#52 0x00007fb45b0b1c53 in server_loop (main_ctx=0x2030110) at src/util/server.c:721
No locals.
#53 0x0000000000407f04 in main (argc=8, argv=<optimized out>) at src/providers/data_provider_be.c:639
opt = <optimized out>
pc = <optimized out>
opt_logger = 0x202d670 "files"
be_domain = 0x202d590 "LDAP"
srv_name = <optimized out>
main_ctx = 0x2030110
confdb_path = <optimized out>
ret = 0
uid = 0
gid = 0
long_options = {{longName = 0x0, shortName = 0 '\000', argInfo = 4, arg = 0x632ac0 <poptHelpOptions@@LIBPOPT_0>, val = 0, descrip = 0x41e317 "Help options:", argDescrip = 0x0}, {longName = 0x41e325 "debug-level", shortName = 100 'd', argInfo = 2, arg = 0x632ba8 <debug_level>, val = 0, descrip = 0x41e331 "Debug level", argDescrip = 0x0}, {longName = 0x41e33d "debug-to-files", shortName = 102 'f', argInfo = 1073741824, arg = 0x632aa4 <debug_to_file>, val = 0, descrip = 0x41ea60 "Send the debug output to files instead of stderr", argDescrip = 0x0}, {longName = 0x41e34c "debug-to-stderr", shortName = 0 '\000', argInfo = 1073741824, arg = 0x632aa0 <debug_to_stderr>, val = 0, descrip = 0x41ea98 "Send the debug output to stderr directly.", argDescrip = 0x0}, {longName = 0x41e35c "debug-timestamps", shortName = 0 '\000', argInfo = 2, arg = 0x632b88 <debug_timestamps>, val = 0, descrip = 0x41e36d "Add debug timestamps", argDescrip = 0x0}, {longName = 0x41e382 "debug-microseconds", shortName = 0 '\000', argInfo = 2, arg = 0x632b90 <debug_microseconds>, val = 0, descrip = 0x41eac8 "Show timestamps with microseconds", argDescrip = 0x0}, {longName = 0x41e399 "logger", shortName = 0 '\000', argInfo = 1, arg = 0x7ffe7d232828, val = 0, descrip = 0x41e395 "Set logger", argDescrip = 0x41e3a0 "stderr|files|journald"}, {longName = 0x41e3b6 "uid", shortName = 0 '\000', argInfo = 2, arg = 0x7ffe7d232820, val = 0, descrip = 0x41eaf0 "The user ID to run the server as", argDescrip = 0x0}, {longName = 0x41e3ba "gid", shortName = 0 '\000', argInfo = 2, arg = 0x7ffe7d232824, val = 0, descrip = 0x41eb18 "The group ID to run the server as", argDescrip = 0x0}, {longName = 0x41f111 "domain", shortName = 0 '\000', argInfo = 1, arg = 0x7ffe7d232830, val = 0, descrip = 0x41eb40 "Domain of the information provider (mandatory)", argDescrip = 0x0}, {longName = 0x0, shortName = 0 '\000', argInfo = 0, arg = 0x0, val = 0, descrip = 0x0, argDescrip = 0x0}}
__FUNCTION__ = "main"
More information about the samba-technical
mailing list