make test spin / Having trouble debugging samba on FreeBSD (threads)

[Andrew Bartlett]

On Mon, 2012-10-01 at 12:30 +0200, Volker Lendecke wrote:
> On Sun, Sep 30, 2012 at 09:13:34AM +1000, Andrew Bartlett wrote:
> > I've been for some time puzzled as to why the build farm fails quite so
> > many tests, with name resolution errors.
> > 
> > I first thought it was to do with samba_dnsupdate not locking the
> > dns_hosts_file, so I reworked that code to have much more certain
> > locking behaviour.
> > 
> > Noticing that one of the failing platforms is FreeBSD, I've reproduced
> > it here on 8.2.  When I run make test or make testenv I get: 
> > 
> > /usr/local/samba-build/source4/scripting/bin/samba_dnsupdate: Traceback
> > (most recent call last):
> > /usr/local/samba-build/source4/scripting/bin/samba_dnsupdate:   File
> > "/usr/local/samba-build/source4/scripting/bin/samba_dnsupdate", line
> > 507, in <module>
> > /usr/local/samba-build/source4/scripting/bin/samba_dnsupdate:
> > get_credentials(lp)
> > /usr/local/samba-build/source4/scripting/bin/samba_dnsupdate:   File
> > "/usr/local/samba-build/source4/scripting/bin/samba_dnsupdate", line
> > 121, in get_credentials
> > /usr/local/samba-build/source4/scripting/bin/samba_dnsupdate:
> > creds.get_named_ccache(lp, ccachename)
> > /usr/local/samba-build/source4/scripting/bin/samba_dnsupdate:
> > RuntimeError: kinit for LOCALDC$@SAMBA.EXAMPLE.COM failed (Cannot
> > contact any KDC for requested realm)
> > 
> > For some reason the kdc, cldapd, dns and echo tasks spin:
> > 
> > /usr/local/samba-build/source4/scripting/bin/samba_dnsupdate: 
> > 97782   2  S+     0:00.00 samba: task[dcesrv] (samba)
> > 97783   2  I+     0:00.01 samba: task[nbtd] (samba)
> > 97784   2  S+     0:00.01 samba: task[wreplsrv] (samba)
> > 97785   2  S+     0:00.01 samba: task[ldapsrv] (samba)
> > 97786   2  R+     0:15.38 samba: task[cldapd] (samba)
> > 97787   2  R+     0:13.64 samba: task[kdc] (samba)
> > 97788   2  S+     0:00.04 samba: task[dreplsrv] (samba)
> > 97789   2  S+     0:00.00 samba: task[winbind] (samba)
> > 97790   2  S+     0:00.00 samba: task[ntp_signd] (samba)
> > 97791   2  I+     0:00.09 samba: task[kccsrv] (samba)
> > 97792   2  S+     0:00.01 samba: task[dnsupdate] (samba)
> > 97793   2  R+     0:15.75 samba: task[dns] (samba)
> > 97794   2  R+     0:15.58 samba: task[echo] (samba)
> > 97795   2  S+     0:00.00 samba: task[smbsrv] (samba)
> > 
> > What I need help with is running gdb on samba.  I know it sounds silly,
> > but neither gdb, nor gdb72 will attach to the process (gdb -p <PID>),
> > each giving internal assertions on threads.  Is there a different
> > debugger I should use, or should I try a newer FreeBSD than 8.2?  I
> > already tried compiling with --disable-pthredpool.
> > 
> > Volker:
> > 
> > Does the full (top level) 'make test' pass for you in your FreeBSD
> > development environment?
> 
> No, it does not. I have spinning samba processes as well.
> They are spinning in a loop:

> My impression is that we are looping due to the
> non-blocking optimization in tsocket_bsd.c:845. The kernel
> gives us "0 bytes available" as per FIONREAD, and we then
> successfully recvfrom those 0 bytes from the socket. And as
> this is not a valid DNS/KRB/CLDAP/... packet, we enter that
> loop immediately again. Maybe we should re-think the
> handling of "0" bytes available in
> tdgram_bsd_recvfrom_handler.

Thanks Volker for the excellent analysis.

I've tried metze's patch (lib/tsocket: fix loop in tdgram_bsd_recvfrom()
(bug #9184) at
https://gitweb.samba.org/?p=metze/samba/wip.git;a=commitdiff;h=730d4e5e7c66f4f296e7e4c2919ce9ce53370792), and I think it helps, because I now get further and fail with:

/usr/local/samba-build/bin/smbd: Abort trap (core dumped)
/usr/local/samba-build/bin/smbd: 127.0.0.11 (ipv4:127.0.0.11:57599)
connect to service print3 initially as user PLUGINDOMAIN/Administrator
(uid=0, gid=20) (pid 97599)
/usr/local/samba-build/bin/smbd: Failed to set gid privileges to (20,20)
now set to (0,20) uid=(0,0)
/usr/local/samba-build/bin/smbd: PANIC (pid 97684): failed to set gid
/usr/local/samba-build/bin/smbd: 
/usr/local/samba-build/bin/smbd: unable to produce a stack trace on this
platform
/usr/local/samba-build/bin/smbd: smb_panic(): calling panic action
[/usr/local/samba-build/selftest/gdb_backtrace 97684]
/usr/local/samba-build/bin/smbd: Abort trap (core dumped)

So, I'm back where I started, that is, being unable to debug with gdb on
FreeBSD 8.2.  I'm installing FreeBSD 9.0 to see if this helps provide a
working threads-aware debugger.

Andrew Bartlett

-- 
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org