[Samba] Overloaded samba server. Is it a bug?

[Andrew Bartlett]

On Wed, 2005-11-02 at 18:53 -0300, Martin wrote:
> On Monday 31 October 2005 18:27, Andrew Bartlett wrote:
> > > >Now we are testing this configuration and waiting for the results.
> > >
> > > Bad luck... the load exploited again :(
> >
> > My gut feeling is that this is an e-Directory bug, but have you posted
> > logfiles and traffic somewhere?  (Warning: unencrypted LDAP traffic will
> > include passwords).
> >
> 
>  Dear  Andrew:
>     
>     We've been trying to test our environment as recommended in the last 
> posts. We switched the backend to openLDAP trying to discard the idea of 
> a bug in eDirectory. And here... the conclusion:
>     
>     
>     Backend switch:
>     
>     
>     1) Sadly, Samba server still getting overloaded. The server doesn't 
> hang as in the previous scenario, but it gets extremely slow and there's 
> no way to provide service with it (load grows up to 60 or 70). It stops 
> responding a couple of minutes after the load gets to the limit.
>     
>     2) There's an incredible amount of smbd childs in "D" state 
> (uninterruptable sleep), when the load starts to raise. It happens with 
> both backends (it's softer with openLDAP, but still unusable).

This is a *very* important clue.  If this were an LDAP issue, then Samba
should be in S state, and the ldap processes should be going nuts.  

>     3) The number of sleeping processes is considerably lower with openLDAP.
>     
>     It seems that, something is beating the samba server because of a 
> bug perhaps, or a misconfiguration. The system is a little (but not 
> much) tolerant when openLDAP is used as backend (instead of eDir), but 
> the problems still no matter the directory service being used.
>     What do you think about a client triggering this behaviour some way?

I now suspect the LDAP angle is a red herring, and I'm instead thinking
'kernel issue'.  

>     
>     Weird things found:
>     
>     I'll comment some lines about a couple of strange things i saw. They 
> may be completely unrelated to the main problem, but here they go just 
> in case.
>     
>     1) Some times (according to what an strace attached to the parent 
> smbd process shows us), a user working on an XLS file starts a curious 
> behaviour in which the server tries to find a file that no longer exist 
> in a periodically basis (i.e: loop). We think the user deleted the file, 
> still an smbd process kept trying to access it. (it was complaining with 
> "file does not exist" messages permanently) (A few minutes later when the loop 
> was happening we went to the user desktop and found out he has already turned 
> off his machine!)
>     
>     We've captured service logs, straces, ps aux snapshots during the 
> load issue and a couple of lsofs.  (The whole samba's logs are more than 1G 
> and is impossible to determinate a fail, because the server still responding 
> until the load is too high to continue serving files)
>     
>     Is there any way to get some more verbosity? any different way of 
> debugging (gdb maybe?)?
    
What would be interesting is to find out where each of those smbd
processes is waiting.  Ie, what call is causing the kernel to put the
process in D state.  Is it the same call, or a lot of different calls? 

Given the load problems, I presume the processes are chewing CPU time?

Andrew Bartlett

-- 
Andrew Bartlett                                http://samba.org/~abartlet/
Samba Developer, SuSE Labs, Novell Inc.        http://suse.de
Authentication Developer, Samba Team           http://samba.org
Student Network Administrator, Hawker College  http://hawkerc.net
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://lists.samba.org/archive/samba/attachments/20051103/53da4890/attachment.bin