[Samba] samba4 ldap high load and port queue overflow
Andrew Bartlett
abartlet at samba.org
Wed Nov 25 07:59:18 UTC 2015
On Tue, 2015-11-24 at 17:48 +0300, Yuriy Tabolin wrote:
> Thanks for answer. I am very sad to see that.
> I had examined samba behavior with tracer and had seen too many poll,
> fcntl and stat system calls by samba task[ldapsrv] process
>
> Elapsed Times for PID 1423,
>
> SYSCALL TIME (ns)
> dup 2653
> flock 5811
> lseek 33513
> geteuid 42747
> getpid 63819
> getpeername 85480
> getsockname 151683
> socket 237632
> accept 311871
> fstat 404420
> madvise 452120
> readv 601524
> mmap 858391
> writev 910337
> gettimeofday 929595
> read 1064811
> close 1270401
> open 1733182
> lstat 1938394
> bind 1939462
> unlink 2074873
> munmap 2569299
> mkdir 3539965
> clock_gettime 5662586
> stat 7543724
> fcntl 74322331
> poll 5393911015
That poll() count is very high, I do wonder if something isn't working
right with your system. I would have expected it to be more in line
with fcntl(), which we use a lot of in TDB.
If you could profile your use case on a linux host with the perf tools,
that may give us some insights.
> I think may be samba something doing not optimal way. Like this,
> first
> we try to create directory (mkdir), then we check it (stat). May be
> that
> leads to impact filesystems and samba ldap works slower than
> openldap.
>
> fcntl(15,F_SETLKW,0x7fffffffd118) = 0 (0x0)
> fcntl(15,F_SETLKW,0x7fffffffd188) = 0 (0x0)
> clock_gettime(13,{1448375653.000000000 }) = 0 (0x0)
> mkdir("/var/db/samba4/private/sam.ldb.d",0700) ERR#17 'File exists'
> stat("/var/db/samba4/private/sam.ldb.d/DC=DOMAINDNSZONES,DC=SPEECHPRO
> ,DC=COM.ldb",{
> mode=-rw------- ,inode=242034,size=4247552,blksize=32768 }) = 0 (0x0)
> fcntl(16,F_SETLKW,0x7fffffffd098) = 0 (0x0)
> fcntl(16,F_SETLKW,0x7fffffffd138) = 0 (0x0)
> fcntl(16,F_SETLKW,0x7fffffffd098) = 0 (0x0)
> fcntl(16,F_SETLKW,0x7fffffffd138) = 0 (0x0)
> fcntl(16,F_SETLKW,0x7fffffffd098) = 0 (0x0)
> fcntl(16,F_SETLKW,0x7fffffffd138) = 0 (0x0)
> clock_gettime(13,{1448375653.000000000 }) = 0 (0x0)
> clock_gettime(13,{1448375653.000000000 }) = 0 (0x0)
> fcntl(16,F_SETLK,0x7fffffffd098) = 0 (0x0)
> fcntl(16,F_SETLKW,0x7fffffffd118) = 0 (0x0)
> fcntl(16,F_SETLKW,0x7fffffffd188) = 0 (0x0)
> clock_gettime(13,{1448375653.000000000 }) = 0 (0x0)
> mkdir("/var/db/samba4/private/sam.ldb.d",0700) ERR#17 'File exists'
> stat("/var/db/samba4/private/sam.ldb.d/DC=FORESTDNSZONES,DC=SPEECHPRO
> ,DC=COM.ldb",{
> mode=-rw------- ,inode=242035,size=4247552,blksize=32768 }) = 0 (0x0)
> fcntl(17,F_SETLKW,0x7fffffffd098) = 0 (0x0)
> fcntl(17,F_SETLKW,0x7fffffffd138) = 0 (0x0)
> fcntl(17,F_SETLKW,0x7fffffffd098)
>
These really should happen only once per startup of the process, how
often do you see it?
> 23.11.2015 18:44, mathias dufresne пишет:
> > Hi,
> >
> > I feel like I must reply off-list: Samba 4 is not yet able to use
> > threads.
> > To work around that we are about to deploy our DCs as virtual
> > machines, to be able to multiply the number of Samba running.
> >
> > I expect you would have already thought about that, but in case you
> > didn't, I said it :p
> >
> > I will follow closely that topic you opened. Sorry to not help
> > more.
> >
> > Cheers,
> >
> > mathias
> >
> >
> >
> > 2015-11-23 14:50 GMT+01:00 Yuriy Tabolin <tabolin at speechpro.com
> > <mailto:tabolin at speechpro.com>>:
> >
> > Hi all.
> > I have samba 4.2.3 on freebsd 10.1 server. There are three DC
> > and
> > about 350 PC on domain. DCs have 3 CPU and 3GB RAM. Some
> > servers
> > with services like apache, exim, dovecot, etc use samba4 ldap
> > (port 389) for user authentication. Some times ago after adding
> > some services to use ldap I found, that samba4 cannot serve all
> > ldap requests. Every 10-30 minutes I see in DCs logs:
> > dc1 kernel: sonewconn: pcb 0xfffff800753d6ab8: Listen queue
> > overflow: 16 already in queue awaiting acceptance (28
> > occurrences)
> >
> > After that I have used tcpdump for recording ldap-traffic and
> > have
> > seen that after TCP handshaking, server some times suddenly
> > send
> > TCP-RST to close connection. I have enlarged DCs resources (CPU
> > and RAM), kern.ipc.somaxconn, did some other system tuning but
> > all that didn't help. Load average on DCs permanently near 0.9
> > -1.0
> > and samba cannot serve all ldap conncetions. ldap clients works
> > well because they use as minimum as two domain controllers as
> > ldap
> > servers. Is there a some performance problem in samba4, slow
> > processing ldap requests or something else?
> > Thanks for any help.
> >
> > --
> > With best regards,
> >
> > Tabolin Yuriy
> > System administrator
> > Speech Technology Center
> >
> >
> > --
> > To unsubscribe from this list go to the following URL and read
> > the
> > instructions: https://lists.samba.org/mailman/options/samba
> >
> >
>
>
> --
> With best regards,
>
> Tabolin Yuriy
> System administrator
> Speech Technology Center
>
--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba
More information about the samba
mailing list