Samba 3.0.32 under Linux: no core file dumped on dump_core() due changing effective gid (DUMPABLE flag is cleared)

[Volodymyr Khomenko]

Hi samba-technical,

I've found the issue inside samba for Linux: it doesn't write coredump file on panic (as specifien in logs),
abort() system call in dump_core() doesn't produce required core ('ulimit -c unlimited' is set).
I've tried also by 'kill -ABRT <smbd_worker_pid>', it doesn't produce the coredump.

Then I tried to insert explicit 'abort()' call to samba code. From the very beginning it dumps, but afterwards
it stops dumping. I've located the point of loosing dumping ability:

main->init_guest_info->make_new_server_info_guest->make_server_info_sam->pdb_enum_group_memberships->
pdb_default_enum_group_memberships->getgroups_unix_user->sys_getgrouplist->
getgrouplist_internals->set_effective_gid->setresgid(-1,gid,-1); with gid different from current (gid=99 for my case).

I see that it's known issue for Linux (I see comments in source/lib/fault.c:dump_core_setup before prctl(PR_SET_DUMPABLE, 1) and
in source/lib/system.c:set_process_capability before prctl(PR_SET_KEEPCAPS, 1) ), and samba is going to use work-around for it.
But still DUMPABLE flag is cleared by setresgid and not restored back (dump_core_setup is called only once).

To make hot-fix, I've put prctl(PR_SET_DUMPABLE, 1) just after problematic line (see my patch below).
But it can be just 'quick and dirty' approach, so proper work-around should be activated for such case.

Could you please describe implemented steps of work-around approach for this Linux-specific behaviour?
Why doesn't it work for our case? Does this work-around present in samba 3.0.32?

--- source/lib/util_sec.c
+++ source/lib/util_sec.c
@@ -226,10 +226,22 @@ void set_effective_uid(uid_t uid)
  Set *only* the effective gid.
  we want to end up with rgid==0 and egid==gid
 ****************************************************************************/
+#include <sys/prctl.h>
 void set_effective_gid(gid_t gid)
 {
 #if USE_SETRESUID
        setresgid(-1,gid,-1);
+       /* Exanet: we lose DUMPABLE flag on switching to another effective gid.
+               We have to restore it to take into account coredump requirement
+               (configured by 'ulimit -c') */
+       {
+               static gid_t prev_effective_gid = -1; /* local static var to track changes only, not each call */
+               if(gid != prev_effective_gid) {
+                       prctl(PR_SET_DUMPABLE, 1);
+                       DEBUG(6, ("setresgid to gid=%d, thus do PR_SET_DUMPABLE again\n", gid));
+                       prev_effective_gid = gid;
+               }
+       }
 #endif

 #if USE_SETREUID

Volodymyr Khomenko,
Software Engineer - Core Technologies
Exanet Ltd.