Summary of flapping tests on sn-devel

Stefan Metzmacher metze at samba.org
Fri Mar 11 12:03:59 UTC 2016


Hi Douglas,

thanks for the summary!

> There is a little script at
> http://git.catalyst.net.nz/gitweb?p=samba-cloud-autobuild.git;a=blob;f=parse-email/parse-autobuild-email;hb=refs/heads/heat
> that reads a mailbox directory containing autobuild failure emails,
> collects the logs from the server, and prints counts of the common
> failure messages. You can restrict it to recent failures, as in this
> example where it shows the 8 worst offenders since a month ago:
> 
> $ ./parse-email/parse-autobuild-email  --since 2016-02-11 -c 8
> found 97 lines matching '^UNEXPECTED' in 125 files matching 'samba.stdout$'
>   41 UNEXPECTED(failure): samba4.winbind.struct.domain_info(ad_member:local)

It's sad, as I spend hours on debugging this (without success),
but I think we should mark this as flapping,

>   11 UNEXPECTED(failure): samba4.ldap.notification.python(ad_dc_ntvfs).__main__.LDAPNotificationTest.test_max_search(ad_dc_ntvfs)

I just posted a fix for that.

>    7 UNEXPECTED(failure): samba4.blackbox.dbcheck.release-4-1-0rc3.check_expected_after_dup_values(none)

I think that's the local-attid problem andrew was about to fix, right?

>    4 UNEXPECTED(failure): samba3.unix.whoami machine account.whoami(nt4_member:local)
>    3 UNEXPECTED(failure): samba3.blackbox.smbclient_s3.plain (ad_member) member creds.sending a message to the remote server(ad_member)
>    3 UNEXPECTED(failure): samba3.wbinfo_simple.(ad_member:local).--krb5auth=SAMBADOMAIN/Administrator%locDCpass1.wbinfo(ad_member:local)
>    3 UNEXPECTED(failure): samba3.rpc.spoolss.printer.addprinter.sd(ad_dc)
>    2 UNEXPECTED(failure): samba4.rpc.backupkey with seal.backupkey.unable_to_decrypt_secret(ad_dc_ntvfs)

That's the one you (or Garming?) tried to fix a few days ago, right?

I've also (hopefully) fixed the new problem with python 2.6:
[1804(11085)/1902 at 1h57m2s]
samba4.ldap.sort.python(ad_dc_ntvfs)(ad_dc_ntvfs)
Traceback (most recent call last):
  File
"/memdisk/autobuild/fl/b3372584/samba/source4/dsdb/tests/python/sort.py", line
8, in <module>
    from collections import Counter
ImportError: cannot import name Counter
command: python
/memdisk/autobuild/fl/b3372584/samba/source4/dsdb/tests/python/sort.py
$SERVER -U"$USERNAME%$PASSWORD" --workgroup=$DOMAIN $LOADLIST  2>&1  |
/memdisk/autobuild/fl/b3372584/samba/selftest/filter-subunit
--fail-on-empty --prefix="samba4.ldap.sort.python(ad_dc_ntvfs)."
--suffix="(ad_dc_ntvfs)"
expanded command: python
/memdisk/autobuild/fl/b3372584/samba/source4/dsdb/tests/python/sort.py
localdc -U"Administrator%locDCpass1" --workgroup=SAMBADOMAIN $LOADLIST
2>&1  | /memdisk/autobuild/fl/b3372584/samba/selftest/filter-subunit
--fail-on-empty --prefix="samba4.ldap.sort.python(ad_dc_ntvfs)."
--suffix="(ad_dc_ntvfs)"
ERROR: Testsuite[samba4.ldap.sort.python(ad_dc_ntvfs)(ad_dc_ntvfs)]
REASON: Exit code was 1

See
https://git.samba.org/autobuild.flakey.sn-devel-104/2016-03-11-1019/samba.stdout

I've attached all patches including the ldap.notification change.

Please have a look.

Can we also get someone else to review and push these?

Thanks!
metze
-------------- next part --------------
From 4efdb40c4b76bc24d7fed7edb8de23466fb3c717 Mon Sep 17 00:00:00 2001
From: Stefan Metzmacher <metze at samba.org>
Date: Fri, 11 Mar 2016 10:16:27 +0100
Subject: [PATCH 1/3] s4:dsdb/test/notification: make test_invalid_filter more
 resilient against ordering races

We saw a lot of flapping tests with:

    [1793(11038)/1892 at 1h55m26s]
    samba4.ldap.notification.python(ad_dc_ntvfs)(ad_dc_ntvfs)
    UNEXPECTED(failure):
    samba4.ldap.notification.python(ad_dc_ntvfs).__main__.LDAPNotificationTest.test_max_search(ad_dc_ntvfs)
    REASON: Exception: Exception: Traceback (most recent call last):
      File
    "/memdisk/autobuild/fl/b1782183/samba/source4/dsdb/tests/python/notification.py",
    line 181, in test_max_search
        self.assertEquals(num, ERR_TIME_LIMIT_EXCEEDED)
    AssertionError: 11 != 3

Signed-off-by: Stefan Metzmacher <metze at samba.org>
---
 source4/dsdb/tests/python/notification.py | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/source4/dsdb/tests/python/notification.py b/source4/dsdb/tests/python/notification.py
index d799c91..a6dadfa 100755
--- a/source4/dsdb/tests/python/notification.py
+++ b/source4/dsdb/tests/python/notification.py
@@ -168,6 +168,8 @@ delete: otherLoginWorkstations
                                                    attrs=["name"],
                                                    controls=["notification:1"],
                                                    timeout=1)
+        num_admin_limit = 0
+        num_time_limit = 0
         for i in xrange(0, max_notifications + 1):
             try:
                 for msg in notifies[i]:
@@ -175,10 +177,15 @@ delete: otherLoginWorkstations
                 res = notifies[i].result()
                 self.fail()
             except LdbError, (num, _):
-                if i >= max_notifications:
-                    self.assertEquals(num, ERR_ADMIN_LIMIT_EXCEEDED)
-                else:
-                    self.assertEquals(num, ERR_TIME_LIMIT_EXCEEDED)
+                if num == ERR_ADMIN_LIMIT_EXCEEDED:
+                    num_admin_limit += 1
+                    continue
+                if num == ERR_TIME_LIMIT_EXCEEDED:
+                    num_time_limit += 1
+                    continue
+                raise
+         self.assertEqual(num_admin_limit, 1)
+         self.assertEqual(num_time_limit, max_notifications)
 
     def test_invalid_filter(self):
         """Testing invalid filters for notifications"""
-- 
1.9.1


From 1741aac7f4b426dadc99bc307825f03abb092904 Mon Sep 17 00:00:00 2001
From: Stefan Metzmacher <metze at samba.org>
Date: Fri, 11 Mar 2016 10:39:13 +0100
Subject: [PATCH 2/3] s4:dsdb/test/sort: avoid 'from collections import
 Counter'

This is only available in python 2.7 and >= 3.1

This should fix make test with python 2.6.

Signed-off-by: Stefan Metzmacher <metze at samba.org>
---
 source4/dsdb/tests/python/sort.py | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/source4/dsdb/tests/python/sort.py b/source4/dsdb/tests/python/sort.py
index c4d2c44..b7b9d83 100644
--- a/source4/dsdb/tests/python/sort.py
+++ b/source4/dsdb/tests/python/sort.py
@@ -5,7 +5,6 @@ from unicodedata import normalize
 import locale
 locale.setlocale(locale.LC_ALL, ('en_US', 'UTF-8'))
 
-from collections import Counter
 import optparse
 import sys
 import os
@@ -191,7 +190,13 @@ class BaseSortTests(samba.tests.TestCase):
                 self.expected_results[k] = (fixed, list(reversed(fixed)))
         for k in ('streetAddress', 'postalAddress'):
             if k in self.expected_results:
-                c = Counter([u[k] for u in self.users])
+                c = {}
+                for u in self.users:
+                    x = u[k]
+                    if x in c:
+                        c[x] += 1
+                        continue
+                    c[x] = 1
                 fixed = []
                 for x in FIENDISH_TESTS:
                     fixed += [norm(x)] * c[x]
-- 
1.9.1


From 1a2612b5fb330b8c84ca78fb3282074ec4fd4076 Mon Sep 17 00:00:00 2001
From: Stefan Metzmacher <metze at samba.org>
Date: Fri, 11 Mar 2016 10:49:21 +0100
Subject: [PATCH 3/3] selftest: mark
 samba4.winbind.struct.domain_info.ad_member as flapping

See https://lists.samba.org/archive/samba-technical/2016-March/112861.html

  found 517 lines matching '^UNEXPECTED' in 641 files matching 'samba.stdout$'
   175 UNEXPECTED(failure): samba4.winbind.struct.domain_info(ad_member:local)
    19 UNEXPECTED(failure): samba4.winbind.struct.domain_info(s3member:local)
    12 UNEXPECTED(failure): samba4.rpc.backupkey with seal.backupkey.server_wrap_encrypt_decrypt_wrong_key(ad_dc_ntvfs)
    12 UNEXPECTED(failure): samba4.drs.delete_object.python(promoted_dc).delete_object.DrsDeleteObjectTestCase.test_ReplicateDeletedObject1(promoted_dc)
    12 UNEXPECTED(failure): samba4.rpc.backupkey with seal.backupkey.server_wrap_decrypt_wrong_r2(ad_dc_ntvfs)
    11 UNEXPECTED(failure): samba4.ldap.notification.python(ad_dc_ntvfs).__main__.LDAPNotificationTest.test_max_search(ad_dc_ntvfs)

We'll see if we also need to add
samba4.winbind.struct.domain_info.s3member
before we're able to identify and fix the problem.

Signed-off-by: Stefan Metzmacher <metze at samba.org>
---
 selftest/flapping | 1 +
 1 file changed, 1 insertion(+)

diff --git a/selftest/flapping b/selftest/flapping
index 81629f9..2f0ef55 100644
--- a/selftest/flapping
+++ b/selftest/flapping
@@ -29,3 +29,4 @@
 ^samba4.drs.delete_object.python # flakey on sn-devel
 ^samba4.blackbox.samba_tool_demote # flakey on sn-devel
 ^samba4.smb2.create.mkdir-dup\(ad_dc_ntvfs\) # This test (for bug 11486) involves a race, not always protected against in the NTVFS file server
+^samba4.winbind.struct.domain_info.ad_member # flakey on sn-devel-104 and sn-devel-144
-- 
1.9.1

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20160311/18a20450/signature.sig>


More information about the samba-technical mailing list