s4: Slow access to group membership for 4.0.0alpha12-GIT-77b9b97

[Michael Wood]

Hi

I've been using Samba 4 for authentication for several months now and
it's been working really well.  Only a couple of OS X servers joined
to the domain.

Yesterday I added around 34000 users to Samba 4
(4.0.0alpha12-GIT-77b9b97) and was adding all of them into the same
group.  Before that there were around 500 users and everything had
been working fine since I installed Samba.  I have not felt the need
to upgrade, since it was working well.

I was using a Python script and adding the users via python-ldap's
add_s().  This took much longer than expected.  I think next time I'll
try ldbadd from the command line instead.

I was looping through the users, adding a user, adding them to the
group and then moving on to the next user.  Adding the user to the
group was taking much longer than the already slow process of creating
the user, so I decided to just add the users first and to the group
add separately.

If I tried to do these at the same time, it seems that they blocked
each other.  i.e. the user creation script would pause for a while and
then continue.  I was adding 100 members at a time in the group add
script, also via python-ldap.

While doing this I discovered that I could use kinit to authenticate
users, but looking up group membership took incredibly long or just
failed.  Running "id" or "groups" from two different OS X boxes joined
to the domain sometimes gave took a long time and then returned just
group numbers and no group names, or sometimes seemed to have only
some of the groups.  There also seems to be a bug in "groups" on OS X,
since it sometimes crashed immediately complaining about a double free
or something.  I don't have the error message at the moment.

Running top on the Samba box showed that one of the samba processes
started using up 100% of the CPU while I was running "id" or "groups"
on one of the OS X boxes.  strace showed what looked like thousands of
locks.  Unfortunately I don't have the details now, because it was
late and I needed to get it working again ASAP, but I think it was
lots of fcntl() calls.  I'll try to get the details by investigating
on another box.  To fix things I restored a backup from before I
started to add the users.

Obviously it's an oldish version of Samba.  Could this be the problem?
 Otherwise maybe some sort of indexing issue?

I'll try to reproduce the problem on another machine and I'll also try
upgrading Samba on the test machine to see if that improves the
situation.

Thanks.

-- 
Michael Wood <esiotrot at gmail.com>