fcntl spinlock in Linux?

[Alex Korobkin]

2013/8/15 Volker Lendecke <Volker.Lendecke at sernet.de>

> On Wed, Aug 14, 2013 at 10:52:43AM -0400, Alex Korobkin wrote:
> > Yes, sorry for bad wording. The machine was hung, not crashed.
> >
> > Here is a trace from SysRq L:
> >
> > [363278.604569] Call Trace:
> > [363278.604578]  [<ffffffff8117e570>] lock_flocks+0x10/0x20
> > [363278.604584]  [<ffffffff8117fbc1>] __posix_lock_file+0x41/0x5c0
> > [363278.604590]  [<ffffffff8118033b>] vfs_lock_file+0x3b/0x40
> > [363278.604596]  [<ffffffff8118064f>] fcntl_setlk+0x16f/0x320
> > [363278.604603]  [<ffffffff811493b7>] sys_fcntl+0x167/0x5c0
> > [363278.604609]  [<ffffffff8169e112>] system_call_fastpath+0x16/0x1b
> > [363278.604613] Code: c3 0f 1f 84 00 00 00 00 00 55 b8 00 01 00 00 48 89
> e5
> > f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 74 11 0f 1f 84 00 00 00 00 00 f3
> 90
> > <0f> b6 07 38 c2 75 f7 c9 c3 0f 1f 44 00 00 55 48 89 e5 ff 14 25
> >
> > Machine code quoted there can seemingly be translated back into
> >
> > void lock_flocks(void)
> > {
> >        spin_lock(&file_lock_lock);
> > }
> >
> > in the kernel.
> >
> >
> > > The discussion suggests this patch to try with the kernel:
> > > >
> https://lists.samba.org/archive/samba-technical/2013-January/090224.html
> > > >
> > > > I'm not very confident about patching the kernel, and curious if
> there is
> > > > anything I could try to mitigate it on Samba's side. What would you
> > > > recommend?
> > > >
> > >
> > > The 3.11 kernel will be getting a first round of patches that breaks up
> > > the global file_lock_lock spinlock into a per-inode lock for the most
> > > part, and makes some other scalability improvements. Without knowing
> > > what specific problem you're having I can't really say whether those
> > > changes will help you however.
> > >
> > > I'm also working on a set of patches to help address the thundering
> > > herd problem when a lock is released. That was the main problem that
> > > Volker saw. I have a scheme to address that too and a set of patches,
> > > but it's 3.12 material at best (and probably more like 3.13).
> > >
> > > --
> > > Jeff Layton <jlayton at redhat.com>
> > >
> >
> > I'm attaching a per-process stack trace as well for you to have a look.
>
> Can you get us a few popular user space stack traces under
> load as well? Maybe there are inefficiencies in the tdb use
> in your case that we might fix in Samba itself.
>
>
Here is a similar dump from a printserver running with ~50 smbd processes.
Samba 3.6.18, libtdb 1.2.12, Ubuntu 12.04 x64. Is this what you wanted to
have a look at?


> Volker
>
> --
> SerNet GmbH, Bahnhofsallee 1b, 37081 Göttingen
> phone: +49-551-370000-0, fax: +49-551-370000-9
> AG Göttingen, HRB 2816, GF: Dr. Johannes Loxen
> http://www.sernet.de, mailto:kontakt at sernet.de
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smbd_under_load.gz
Type: application/x-gzip
Size: 34082 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20130816/a887f2ee/attachment.bin>