[Samba] Overloaded samba server. Is it a bug?

[Martin]

On Wednesday 02 November 2005 19:34, Andrew Bartlett wrote:
> On Wed, 2005-11-02 at 18:53 -0300, Martin wrote:
> > On Monday 31 October 2005 18:27, Andrew Bartlett wrote:
> > > > >Now we are testing this configuration and waiting for the results.
> > > >
> > > > Bad luck... the load exploited again :(
> > >
> > > My gut feeling is that this is an e-Directory bug, but have you posted
> > > logfiles and traffic somewhere?  (Warning: unencrypted LDAP traffic
> > > will include passwords).
> >
> >  Dear  Andrew:
> >
> >     We've been trying to test our environment as recommended in the last
> > posts. We switched the backend to openLDAP trying to discard the idea of
> > a bug in eDirectory. And here... the conclusion:
> >
> >
> >     Backend switch:
> >
> >
> >     1) Sadly, Samba server still getting overloaded. The server doesn't
> > hang as in the previous scenario, but it gets extremely slow and there's
> > no way to provide service with it (load grows up to 60 or 70). It stops
> > responding a couple of minutes after the load gets to the limit.
> >
> >     2) There's an incredible amount of smbd childs in "D" state
> > (uninterruptable sleep), when the load starts to raise. It happens with
> > both backends (it's softer with openLDAP, but still unusable).
>
> This is a *very* important clue.  If this were an LDAP issue, then Samba
> should be in S state, and the ldap processes should be going nuts.
>
> >     3) The number of sleeping processes is considerably lower with
> > openLDAP.
> >
> >     It seems that, something is beating the samba server because of a
> > bug perhaps, or a misconfiguration. The system is a little (but not
> > much) tolerant when openLDAP is used as backend (instead of eDir), but
> > the problems still no matter the directory service being used.
> >     What do you think about a client triggering this behaviour some way?
>
> I now suspect the LDAP angle is a red herring, and I'm instead thinking
> 'kernel issue'.
>
> >     Weird things found:
> >
> >     I'll comment some lines about a couple of strange things i saw. They
> > may be completely unrelated to the main problem, but here they go just
> > in case.
> >
> >     1) Some times (according to what an strace attached to the parent
> > smbd process shows us), a user working on an XLS file starts a curious
> > behaviour in which the server tries to find a file that no longer exist
> > in a periodically basis (i.e: loop). We think the user deleted the file,
> > still an smbd process kept trying to access it. (it was complaining with
> > "file does not exist" messages permanently) (A few minutes later when the
> > loop was happening we went to the user desktop and found out he has
> > already turned off his machine!)
> >
> >     We've captured service logs, straces, ps aux snapshots during the
> > load issue and a couple of lsofs.  (The whole samba's logs are more than
> > 1G and is impossible to determinate a fail, because the server still
> > responding until the load is too high to continue serving files)
> >
> >     Is there any way to get some more verbosity? any different way of
> > debugging (gdb maybe?)?
>
> What would be interesting is to find out where each of those smbd
> processes is waiting.  Ie, what call is causing the kernel to put the
> process in D state.  Is it the same call, or a lot of different calls?

How could we find it out? How could we get enough debugging level to reach 
this information?

When the smbd proccess stopped in D state the strace does not show any line...

	--
Mrtn