Autobuild time and flapping tests

[Douglas Bagnall]

On 02/03/18 00:39, Andrew Bartlett via samba-technical wrote:
> On Thu, 2018-03-01 at 11:35 +0100, Andreas Schneider via samba-
> technical wrote:
>> On Thursday, 1 March 2018 11:33:58 CET Volker Lendecke wrote:
>>> On Thu, Mar 01, 2018 at 11:22:04AM +0100, Andreas Schneider wrote:
>>>> 3 autobuilds already failed for me with this patchset. And currently
>>>> autobuild runs for 4-5 hours.

Looking at 
http://git.samba.org/autobuild.flakey.sn-devel-144/2018-02-28-1205/samba.stdout
I was horrified to see the VLV tests taking 16 minutes:

> TOP 10 slowest tests
> samba4.ldap.vlv.python(ad_dc_ntvfs)(ad_dc_ntvfs) -> 981                                                
> samba3.local.nss(ad_dc:local) -> 695
> samba4.ldap.rodc_rwdc.python(rodc)(rodc:local) -> 522
> samba4.drs.link_conflicts.python(promoted_dc)(promoted_dc) -> 409
> samba4.drs.link_conflicts.python(vampire_dc)(vampire_dc) -> 401
> samba4.drs.getncchanges.python(vampire_dc)(vampire_dc) -> 361
> samba4.drs.ridalloc_exop.python(vampire_dc)(vampire_dc) -> 316
> samba3.blackbox.shadow_copy2 NT1 (fileserver)(fileserver) -> 282
> samba4.rpc.samr.passwords.lockout on ncacn_np with (ad_dc_ntvfs) -> 225
> samba4.drs.replica_sync.python(promoted_dc)(promoted_dc:local) -> 221

Looking at historical flakey results,

$ grep -roP '(?<=samba4.ldap.vlv.python\(ad_dc_ntvfs\)\(ad_dc_ntvfs\) -> )\d+' autobuild |sort
[...]
autobuild/git.samba.org+autobuild.flakey.sn-devel-144+2017-12-17-2119+samba.stdout:240
autobuild/git.samba.org+autobuild.flakey.sn-devel-144+2017-12-18-2054+samba.stdout:220
autobuild/git.samba.org+autobuild.flakey.sn-devel-144+2017-12-20-2210+samba.stdout:639
autobuild/git.samba.org+autobuild.flakey.sn-devel-144+2017-12-21-1622+samba.stdout:643
[...]

it seems there was a performance regression in December, and we
strongly suspect the culprit is ef240aaca0ef693a96726ac2366c4542,
"ldb: Intersect the index from SCOPE_ONELEVEL with the index for the
search expression".

The jump from the 600s to 981 is a one-off thing, and seems related to
load on sn-devel.

Douglas