[Samba] winbindd consuming 99% CPU
Sagar Borikar
borikar_sagar at yahoo.com
Sun Nov 9 08:29:07 GMT 2008
Hi
I did some further profiling winbindd and found that there was no network traffic but internally string functions were consuming most of the cpu. Here is the oprofile output of the winbindd which was consuming 99% of cpu
[root at NAS0018f80551d3 new]# opreport -l --demangle=smart "/mnt/RAIDB/vol/new/winbindd" | more
CPU: PMC-Sierra RM9000, speed 0 MHz (estimated) Counted INSTRUCTIONS_ISSUED events (Instructions issued) with a unit mask of 0x00 (No unit mask) count 100000
samples % symbol name
77 20.5882 convert_string
57 15.2406 convert_string_internal
43 11.4973 check_dos_char_slowly
37 9.8930 ascii_push
27 7.2193 smb_iconv
26 6.9519 strwicmp
18 4.8128 toupper_w
15 4.0107 toupper_ascii
11 2.9412 lazy_initialize_conv
10 2.6738 dopr
9 2.4064 format_debug_text
9 2.4064 init_doschar_table
7 1.8717 map_parameter
4 1.0695 dopr_outch
4 1.0695 lp_do_parameter
3 0.8021 fmtstr
2 0.5348 Parameter
2 0.5348 copy_service
2 0.5348 mygetc
1 0.2674 EatComment
1 0.2674 EatWhitespace
1 0.2674 charset_name
1 0.2674 iface_comp
1 0.2674 lp_save_defaults
1 0.2674 lp_string
1 0.2674 set_server_role
1 0.2674 smb_probe_module
1 0.2674 smb_xstrdup
1 0.2674 string_free
1 0.2674 tdb_read
When I backtraced the winbindd through oprofile, here is the call graph which I get,
Counted INSTRUCTIONS_ISSUED events (Instructions issued) with a unit mask of 0x00 (No unit mask) count 100000
samples % image name symbol name
-------------------------------------------------------------------------------
2 7.1429 winbindd convert_string
26 92.8571 winbindd check_dos_char_slowly
77 20.5882 winbindd convert_string
77 65.8120 winbindd convert_string [self]
38 32.4786 winbindd convert_string_internal
2 1.7094 winbindd convert_string
-------------------------------------------------------------------------------
38 100.000 winbindd convert_string
57 15.2406 winbindd convert_string_internal
57 69.5122 winbindd convert_string_internal [self]
15 18.2927 winbindd smb_iconv
10 12.1951 winbindd lazy_initialize_conv
-------------------------------------------------------------------------------
16 32.0000 winbindd check_dos_char_slowly
34 68.0000 winbindd init_doschar_table
43 11.4973 winbindd check_dos_char_slowly
43 50.5882 winbindd check_dos_char_slowly [self]
26 30.5882 winbindd convert_string
16 18.8235 winbindd check_dos_char_slowly
-------------------------------------------------------------------------------
28 100.000 winbindd smb_iconv
37 9.8930 winbindd ascii_push
37 100.000 winbindd ascii_push [self]
-------------------------------------------------------------------------------
15 100.000 winbindd convert_string_internal
27 7.2193 winbindd smb_iconv
28 50.9091 winbindd ascii_push
27 49.0909 winbindd smb_iconv [self]
-------------------------------------------------------------------------------
3 25.0000 winbindd strwicmp
9 75.0000 winbindd map_parameter
26 6.9519 winbindd strwicmp
30 43.4783 winbindd toupper_ascii
26 37.6812 winbindd strwicmp [self]
10 14.4928 libc-2.3.5.so (no symbols)
3 4.3478 winbindd strwicmp
Also pidstat output says that winbindd is consuming more in kernel as well.
Linux 2.6.18 (NAS0018f80551d3) 11/09/08 _mips_
00:27:50 PID %usr %system %guest %CPU CPU Command
00:27:51 31001 45.45 47.47 0.00 92.93 0 winbindd
00:27:52 31001 41.58 56.44 0.00 98.02 0 winbindd
00:27:53 31001 29.00 61.00 0.00 90.00 0 winbindd
00:27:54 31001 33.66 65.35 0.00 99.01 0 winbindd
00:27:55 31001 36.00 54.00 0.00 90.00 0 winbindd
00:27:56 31001 40.40 52.53 0.00 92.93 0 winbindd
00:27:57 31001 34.65 63.37 0.00 98.02 0 winbindd
00:27:58 31001 36.27 52.94 0.00 89.22 0 winbindd
00:27:59 31001 38.00 61.00 0.00 99.00 0 winbindd
Has anybody ever faced similar issue before?
Thanks in advance
Sagar
--- On Wed, 11/5/08, Gerald (Jerry) Carter <jerry at samba.org> wrote:
> From: Gerald (Jerry) Carter <jerry at samba.org>
> Subject: Re: [Samba] winbindd consuming 99% CPU
> To: borikar_sagar at yahoo.com
> Cc: samba at lists.samba.org, kishore_goud at pmc-sierra.com
> Date: Wednesday, November 5, 2008, 7:48 PM
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Sagar Borikar wrote:
> > Hi All,
> >
> > We are currently caught up in a weird situation while
> using samba
> > 3.0.28. There is a requirement from the customer that
> we should support
> > ADS with 64000 objects. When we implemented the
> feature, we found
> > couple of things mentioned below:
> >
> > 1. After joining ADS with 64k users, samba takes
> around 20 - 25 min
> > to populate idmap_cache.tdb and winbindd_cache.tdb
> This is because
> > we haven't turned of enumeration and we would be
> fixing it for
> > next maintenance release of the NAS product.
> >
> > 2. We periodically check the quota state of all the
> users
> > using repquota command. It refers to
> /etc/nsswitch.conf for mapping
> > user ids to names. And winbindd inherently should look
> for its
> > cache as it is populated completely to do the
> conversion.
> >
> > But we observed that every time the repquota command
> is invoked,
> > winbindd starts consuming 98% of CPU. It fetches the
> info from
> > ADS and not from local cache. So we are wondering why
> it doesn't
> > make use of local cache for mapping the ids to names?
>
> Sagar,
>
> The enumeration is killing you performance wise. As a
> workaround
> you could try in creasing the "winbind cache
> time" to something like
> 1 or 2 hours. But there is a constant struggle between
> cache times
> and accuracy.
>
>
>
>
> cheers, jerry
> - --
> =====================================================================
> Samba -------
> http://www.samba.org
> Likewise Software ---------
> http://www.likewisesoftware.com
> "What man is a man who does not make the world
> better?" --Balian
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
> Comment: Using GnuPG with Mozilla -
> http://enigmail.mozdev.org
>
> iD8DBQFJEatJIR7qMdg1EfYRAhmyAJ9CcrZLclYZ+P8q0hHI7EsUt7kRKwCg0km4
> LWFcHLKbEfl8UMYnd4JWqpA=
> =3KW6
> -----END PGP SIGNATURE-----
More information about the samba
mailing list