[Samba] winbind causing huge timeouts/delays since 4.8

Sat Feb 23 21:23:48 UTC 2019

On Sat, 23 Feb 2019 21:54:31 +0100
Alexander Spannagel via samba <samba at lists.samba.org> wrote:

> Am 23.02.19 um 15:48 schrieb Rowland Penny via samba:
> >>>>>>> If you have, as you have, 'files sss winbind' in the the
> >>>>>>> passwd & group line in nsswitch.conf, means this:
> >>>>>>> First /etc/passwd or /etc/group is searched and if the user or
> >>>>>>> group is found, this info is returned.
> >>>>>>> Next sssd will be asked, 'do you know this user or group ?' if
> >>>>>>> found, the info is returned.
> >>>>>>> Finally winbind will be asked, 'do you know this user or
> >>>>>>> group ?' if found, the info is returned.
> >>>>>>>
> >>>>>>> Lets take a user called 'fred', this user is in AD. The first
> >>>>>>> search will return nothing, so sssd is asked, this 'asks' AD
> >>>>>>> and returns the users info. Finally, wait that's it, we have
> >>>>>>> the info, there is no need to ask winbind for anything.
> >>>>>>
> >>>>>> That is incorrect. Alexander stated:
> >>>>>>
> >>>>>>> No. we use max. 3 auth providers: (1. and 2. on all unix
> >>>>>>> servers) 1. unix (local passwd)
> >>>>>>> for static OS/service accounts across all our env
> >>>>>>> 2. sssd (with unix ldap servers as provider)
> >>>>>>> unix experienced user and application related service accounts
> >>>>>>> 3. samba/winbind
> >>>>>>> for windows users/services needing access to a group of unix
> >>>>>>> servers
> >>>>>>
> >>>>>> And:
> >>>>>>
> >>>>>>> They don't - as stated above we use sssd for query/caching
> >>>>>>> entries from our ldap directory server and not Windows
> >>>>>>> DomainConmtrollers
> >>>>>>> - also this is possible, but makes more trouble and don't
> >>>>>>> provide what samba's smb/windbind does.
> >>>>>>
> >>>>>> He clearly writes (in multiple emails) that sssd is configured
> >>>>>> to use his unix ldap servers and not AD.
> >>>>>>
> >>>>>> Maybe three sources of user databases is not regular, but I
> >>>>>> fail to see why this should be a problem (provided that
> >>>>>> usernames, uidNumbers and such are unique across the
> >>>>>> databases).
> >>>>>
> >>>>> And there is the problem, if 'fred' is in /etc/passwd, that user
> >>>>> will be used, but what if you meant fred in ldap or AD ?
> >>>>>
> We are aware of this possible clash and it's handled during users 
> account creation.

If you only used one database, you would be 100% sure you wouldn't get
any clashes.

> 
> >>>>> There is absolutely no point in having 4 databases (yes there
> >>>>> are 4, Unix, sssd, winbind and the ldap lines in smb.conf), they
> >>>>> could all be combined in AD.
> No it won't work as our windows team doesn't accept schema changes
> for unix in AD.

How shall I put this, your windows team is either not telling you the
entire truth or is stupid. The entire RFC2307 attributes are part of
the standard Windows schema. what isn't added is IDMU and this just
makes the Unix attributes tab work in ADUC. You can however use other
tools to create Unix users etc.

> 
> >>>>>
> >>>>> The main problem is that the OP wants Samba changing to cope
> >>>>> with his mess, it might be a valid change, but the reason for
> >>>>> the change is invalid.
> The intial reason we hitted this after upgrade from samba-4.7 to 4.8
> is a script that frequentyl checks the system for changes and a final 
> "chown root.wheel FILE" freezes the system for approx a minute
> (simliar to "wbinfo -i foo"). The winbind and also sssd log showd
> that both were asked about a user "root.wheel" which is another
> question, why the notation which usually (under linux) indicates
> user.group and not an account with a dot in it's name - but more to
> glibc related. Removing winbind from nss fixed the freeze, but isn't
> an option. It leads to the point that asking winbind for an uknown
> user without domain took a long time before it returns
> WBC_ERR_DOMAIN_NOT_FOUND .

You see, there you have 2 methods checking for the same thing, also the
problem you are referring to has been fixed in the latest Samba (I
think it has also been backported).

> 
> >>>>
> >>>> Well, I think the problem is you _assume_ users are in multiple
> >>>> databases and we just don't know that. I think there is a good
> >>>> change Alexander perfectly knows what he is doing and users are
> >>>> unique across databases.
> >>>>
> >>>> Nevertheless, at some point nss is clearly querying winbind,
> >>>> which means nss did not find the user in either /etc/passwd nor
> >>>> via sssd. In the case that winbind _is_ queried, Alexander is
> >>>> experiencing, like he wrote, 'frequently system hangs/slowness
> >>>> for a couple of seconds' and he observed that winbind is causing
> >>>> this behaviour.
> >>>>
> >>>> So maybe we should set our focus on winbind instead of the
> >>>> multiple database stuff and figure out why it behaves like this
> >>>> since the upgrade from 4.7 to 4.8. I would say we should start
> >>>> with fixing the winbind stuff in smb.conf. Right?
> >>>>
> >>>> -Remy
> >>>>
> >>>>
> >>>> P.S. I am following this thread since I also noticed occasional
> >>>> 'hangs' when the system is querying winbind. This is Samba 4.8.7
> >>>> on FreeBSD 11.2.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>> I am quite prepared to help in getting winbind working correctly,
> >>> but this will require the OP changing their smb.conf considerably
> >>> and removing sssd. We do not support sssd, it is not a Samba
> >>> product (for want of a better name). Samba on a Unix domain member
> >>> is designed around 3 binaries, smbd, nmbd and winbind, the latter
> >>> can do just about anything sssd can do, so why use sssd ? Now you
> >>> say that I am making assumptions, well about this one, probably
> >>> somewhere in the mix there will be Windows domain members and the
> >>> users in ldap are unlikely to be known to them.
> >>
> >> I consider sssd as 'just another' user database, like /etc/passwd
> >> (which Samba apparently does support) and I personally cannot see
> >> any difference there, but I respect your opinion.
> >>
> >> Where is it documented winbind should be the only service which
> >> should be used with nss? If it is not documented, maybe it should.
> > 
> > I am not saying that sssd shouldn't be used, just Samba does not
> > support it. If you want to use sssd, then do so, just don't expect
> > to get help with using it, we don't produce it, so don't know it.
> > What I will say is this, there is no need to use both on a Unix
> > domain member, they both do the same thing.
> >
> I'm with you and don't expect that you support sssd.
> On the otherside windind shouln't require to be the only one in 
> nss-setup as i didn't ever heard, that only a certain amount can be 
> taken into the stack. Before nss_sss we used nss_ldap alongside with 
> nss_winbind without issues.

There is no real problem with using multiple methods in nsswitch.conf,
there is just no real point, the first one to produce  a result wins
and as sssd & winbind do the same thing, you don't need both.

> The only interaction i could imagine if one of the libs in the stack 
> calls the stack from beginning waiting for a certain answer ending up
> in a dead end.
> Following the functions calls of the parse_domain_user function it
> seems to me samba takes care about this with the flag
> LOOKUP_NAME_NO_NSS in the code - only an assumption as programming is
> not my daily business.
> 
> 
> >>
> >> The proof of the pudding is of course, Alexander removing sssd from
> >> nsswitch.conf and show us the problem still exists, or better yet,
> >> disappeared.
> > 
> > That is what I am trying to get at, if it is a Samba problem, then
> > it will still be there after sssd is removed and the smb.conf is
> > fixed.
> > 
> >>
> >>
> >>> I have seen quite a few Samba setups that are like this one,
> >>> bending Samba to do something it isn't designed to do, you then
> >>> get complaints that it slow, hangs etc. Probably fixing the set up
> >>> would stop all these problems.
> >>>
> >>> The OP says that it is sssd that is doing the ldap lookups, yet he
> >>> has these in smb.conf:
> >>>
> >>> ldap connection timeout = 10
> >>> ldap timeout = 30
> >>
> >> Yep, these lines should be removed.
> > 
> > Glad you agree ;-)
> > 
> These were a left over from testing config changes to find a solution
> to the problem and reomving them so they get to it's default didn't
> has a real affect:
> [root at centos7dev64 ~]# testparm -v 2>/dev/null </dev/null|grep 
> "ldap.*timeout"
>          ldap connection timeout = 10
>          ldap timeout = 30
> [root at centos7dev64 ~]# time wbinfo -i foo
> failed to call wbcGetpwnam: WBC_ERR_WINBIND_NOT_AVAILABLE
> Could not get info for user foo
> 
> real    1m1.700s
> user    0m0.049s
> sys     0m0.010s
> [root at centos7dev64 ~]# vi /etc/samba/smb.conf
> [root at centos7dev64 ~]# systemctl restart smb winbind sssd ; sss_cache
> -E ; net cache flush
> [root at centos7dev64 ~]# testparm -v 2>/dev/null </dev/null|grep 
> "ldap.*timeout"
>          ldap connection timeout = 2
>          ldap timeout = 15
> [root at centos7dev64 ~]# time wbinfo -i foo
> failed to call wbcGetpwnam: WBC_ERR_DOMAIN_NOT_FOUND
> Could not get info for user foo
> 
> real    0m59.304s
> user    0m0.051s
> sys     0m0.013s
> 
> Curious is also that wbinfo returns different errors for the same
> call: 1. failed to call wbcGetpwnam: WBC_ERR_WINBIND_NOT_AVAILABLE
> 2. failed to call wbcGetpwnam: WBC_ERR_DOMAIN_NOT_FOUND

This is I believe an artefact of the now fixed problem.

> 
> The first somewhow tells me winbind got stuck or not responding in
> time. The second is more the expected response as foo does not
> provide a domain and "winbind use default domain" is set to it's
> default no - not expected the time it takes to get to this finding.
> 
> >>
> >>
> >>> He also has these:
> >>>
> >>> idmap config * : rangesize = 1000000
> >>> idmap config * : range = 1000000-19999999
> >>> idmap config * : backend = autorid
> >>>
> >>> The '*' domain is meant for the Well Known SIDs and anything
> >>> outside the Samba domain. I would have expected something like
> >>> this:
> >>>
> >>> idmap config * : backend = tdb
> >>> idmap config * : range = 3000-7999
> >>> idmap config OPS : backend = rid
> >>> idmap config OPS : range = 10000-999999
> >>
> >> That should also be fixed.
> >>
> >>
> We use this as we have a multi-domain setup on windows side and this
> is a suggested setup from wiki.samba.org:
> https://wiki.samba.org/index.php/Idmap_config_autorid

Cannot argue with that fact, it is there, but it also says it is meant
to be used with the 'DOMAIN' domain not the '*' domain, looks like I
will have to make that more prominent.

> 
> I'll try to somehow reconfig idmap as you suggested taking care of
> all the trees in our forest and will report back if that changes the
> situation.
> 
> >>> If he wants to use Samba in a supported way, then I am more than
> >>> willing to help.
> >>
> >> Thanks. Now let's hope Alexander is willing to jump some hoops.
> > 
> > I am not holding my breath ;-)
> > 
> > Rowland
> > 
> > 
> Thanks for responding and keep breathing ;)
> 
> Alex
> 