[Patches] Expanded group memberships on boundaries of outgoing trusts (bugs #13299, #13300, #13307)

[Stefan Metzmacher]

Hi Douglas,

>>>>> I'll aim this at the perf testing rig,
>>>>
>>>> The results are shown in the attached chart, which compares
>>>> yesterday's version of the patchset with yesterday's origin/master
>>>> over three runs.
>>>>
>>>> Adding large groups (i.e. adding a group with lots of members in a
>>>> single SamDB.add_ldif() operation) is slower, as is populating an
>>>> existing group using add_ldif. Other operations are the same speed or
>>>> slightly faster.
>>>>
>>>> There is quite a bit of noise when testing with just 3 runs, but the
>>>> consistency across the different tests suggests this is a real change.
>>>
>>> Thanks! I hope the new version doesn't have such an impact.
>>>
>>>> I will try again with thew current head of 
>>>> https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=refs/heads/master3-trusts-ok
>>>
>>> The one that I recently pushed should compile and work.
>>
>> We should have a graph for that shortly.
> 
> This one is more accurate for two reasons: it is over 6 runs not 3,
> and it turns out the earlier version had some failures due to memory
> pressure so that the origin/master there had only one run.
> 
> I would say all the variation in the +/- 5 percent range is noise, but
> there is still some slowdown in adding links in a transaction. Though
> much less than before (~17% vs ~70%).

Thanks!

I think this is still noise, if you look at:

cat expanded-group-membership-results-2.json | \
grep link_users_4k | grep -v batch | \
sed -e 's!.*UserTests!!'
.test_06_04_link_users_4k(ad_dc_ntvfs)": 53.451856,
.test_06_04_link_users_4k(ad_dc_ntvfs)": 50.477726,
.test_06_04_link_users_4k(ad_dc_ntvfs)": 44.837215,
.test_06_04_link_users_4k(ad_dc_ntvfs)": 52.058198,
.test_06_04_link_users_4k(ad_dc_ntvfs)": 44.136606,
.test_06_04_link_users_4k(ad_dc_ntvfs)": 47.96203,
.test_06_04_link_users_4k(ad_dc_ntvfs)": 50.986976,
.test_06_04_link_users_4k(ad_dc_ntvfs)": 48.310361,
.test_06_04_link_users_4k(ad_dc_ntvfs)": 45.980828,
.test_06_04_link_users_4k(ad_dc_ntvfs)": 50.925926,
.test_06_04_link_users_4k(ad_dc_ntvfs)": 45.0255,
.test_06_04_link_users_4k(ad_dc_ntvfs)": 49.302871,

cat expanded-group-membership-results-2.json | \
grep adding_users_after_links_6k | \
sed -e 's!.*UserTests!!'


.test_07_01_adding_users_after_links_6k(ad_dc_ntvfs)": 118.127226,
.test_07_01_adding_users_after_links_6k(ad_dc_ntvfs)": 114.895321,
.test_07_01_adding_users_after_links_6k(ad_dc_ntvfs)": 105.889257,
.test_07_01_adding_users_after_links_6k(ad_dc_ntvfs)": 108.302196,
.test_07_01_adding_users_after_links_6k(ad_dc_ntvfs)": 97.354201,
.test_07_01_adding_users_after_links_6k(ad_dc_ntvfs)": 107.637742,
.test_07_01_adding_users_after_links_6k(ad_dc_ntvfs)": 111.840771,
.test_07_01_adding_users_after_links_6k(ad_dc_ntvfs)": 108.734066,
.test_07_01_adding_users_after_links_6k(ad_dc_ntvfs)": 102.872983,
.test_07_01_adding_users_after_links_6k(ad_dc_ntvfs)": 110.352689,
.test_07_01_adding_users_after_links_6k(ad_dc_ntvfs)": 101.089659,
.test_07_01_adding_users_after_links_6k(ad_dc_ntvfs)": 110.584391,

The even rows are from master and in both cases this produced the
slowest, but also the fastest result.

For this

cat expanded-group-membership-results-2.json | \
grep add_exponentially_diminishing_linked_groups| \
sed -e 's!.*UserTests.test_09_02_!!'
add_exponentially_diminishing_linked_groups(ad_dc_ntvfs)": 14.820678,
add_exponentially_diminishing_linked_groups(ad_dc_ntvfs)": 18.788114,
add_exponentially_diminishing_linked_groups(ad_dc_ntvfs)": 15.977902,
add_exponentially_diminishing_linked_groups(ad_dc_ntvfs)": 18.865089,
add_exponentially_diminishing_linked_groups(ad_dc_ntvfs)": 14.428552,
add_exponentially_diminishing_linked_groups(ad_dc_ntvfs)": 15.975584,
add_exponentially_diminishing_linked_groups(ad_dc_ntvfs)": 12.405595,
add_exponentially_diminishing_linked_groups(ad_dc_ntvfs)": 15.100911,
add_exponentially_diminishing_linked_groups(ad_dc_ntvfs)": 14.670697,
add_exponentially_diminishing_linked_groups(ad_dc_ntvfs)": 15.515323,
add_exponentially_diminishing_linked_groups(ad_dc_ntvfs)": 16.562027,
add_exponentially_diminishing_linked_groups(ad_dc_ntvfs)": 14.584863,

we see some larger numbers.
I guess it's related to the ldb_dn_compare in
"dsdb:extended_dn_store: We need to ignore self references on add
operation".

But I'm not sure what to do about that as it's required to pass
samba4.ldap.secdesc.python

Otherwise I get
UNEXPECTED(error):
samba4.ldap.secdesc.python(ad_dc_ntvfs).__main__.OwnerGroupDescriptorTests.test_130(ad_dc_ntvfs)
REASON: Exception: Exception: Traceback (most recent call last):
  File
"/home/metze/devel/samba/3.X/master3/source4/dsdb/tests/python/sec_descriptor.py",
line 697, in test_130
    class_dn = self.create_schema_class(_ldb)
  File
"/home/metze/devel/samba/3.X/master3/source4/dsdb/tests/python/sec_descriptor.py",
line 99, in create_schema_class
    _ldb.add_ldif(ldif)
  File "bin/python/samba/__init__.py", line 229, in add_ldif
    self.add(msg, controls)
LdbError: (19, 'LDAP error 19 LDAP_CONSTRAINT_VIOLATION -  <000020B5:
Referenced object not found at
../source4/dsdb/samdb/ldb_modules/extended_dn_store.c:360:extended_replace_callback>
<>')

The strange thing is that the resulting object in the database
has a GUID part in the defaultObjectCategory attribute I checked
it in
sam.ldb.d/CN\=SCHEMA\,CN\=CONFIGURATION\,DC\=SAMBA2008R2\,DC\=EXAMPLE\,DC\=COM.ldb
I'm wondering where this got added (and it's not get_parsed_dns()).

I'm also wondering why the following fails against (even older) samba
versions:

dn: CN=SelfReference,CN=Users,DC=addom,DC=samba,DC=example,DC=com
objectClass: group
member: CN=SelfReference,CN=Users,DC=addom,DC=samba,DC=example,DC=com

while it seems to work for the defaultObjectCategory attribute.

metze

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20180301/e99c3002/signature.sig>