Summary of flapping tests on sn-devel
Stefan Metzmacher
metze at samba.org
Fri Mar 11 12:03:59 UTC 2016
Hi Douglas,
thanks for the summary!
> There is a little script at
> http://git.catalyst.net.nz/gitweb?p=samba-cloud-autobuild.git;a=blob;f=parse-email/parse-autobuild-email;hb=refs/heads/heat
> that reads a mailbox directory containing autobuild failure emails,
> collects the logs from the server, and prints counts of the common
> failure messages. You can restrict it to recent failures, as in this
> example where it shows the 8 worst offenders since a month ago:
>
> $ ./parse-email/parse-autobuild-email --since 2016-02-11 -c 8
> found 97 lines matching '^UNEXPECTED' in 125 files matching 'samba.stdout$'
> 41 UNEXPECTED(failure): samba4.winbind.struct.domain_info(ad_member:local)
It's sad, as I spend hours on debugging this (without success),
but I think we should mark this as flapping,
> 11 UNEXPECTED(failure): samba4.ldap.notification.python(ad_dc_ntvfs).__main__.LDAPNotificationTest.test_max_search(ad_dc_ntvfs)
I just posted a fix for that.
> 7 UNEXPECTED(failure): samba4.blackbox.dbcheck.release-4-1-0rc3.check_expected_after_dup_values(none)
I think that's the local-attid problem andrew was about to fix, right?
> 4 UNEXPECTED(failure): samba3.unix.whoami machine account.whoami(nt4_member:local)
> 3 UNEXPECTED(failure): samba3.blackbox.smbclient_s3.plain (ad_member) member creds.sending a message to the remote server(ad_member)
> 3 UNEXPECTED(failure): samba3.wbinfo_simple.(ad_member:local).--krb5auth=SAMBADOMAIN/Administrator%locDCpass1.wbinfo(ad_member:local)
> 3 UNEXPECTED(failure): samba3.rpc.spoolss.printer.addprinter.sd(ad_dc)
> 2 UNEXPECTED(failure): samba4.rpc.backupkey with seal.backupkey.unable_to_decrypt_secret(ad_dc_ntvfs)
That's the one you (or Garming?) tried to fix a few days ago, right?
I've also (hopefully) fixed the new problem with python 2.6:
[1804(11085)/1902 at 1h57m2s]
samba4.ldap.sort.python(ad_dc_ntvfs)(ad_dc_ntvfs)
Traceback (most recent call last):
File
"/memdisk/autobuild/fl/b3372584/samba/source4/dsdb/tests/python/sort.py", line
8, in <module>
from collections import Counter
ImportError: cannot import name Counter
command: python
/memdisk/autobuild/fl/b3372584/samba/source4/dsdb/tests/python/sort.py
$SERVER -U"$USERNAME%$PASSWORD" --workgroup=$DOMAIN $LOADLIST 2>&1 |
/memdisk/autobuild/fl/b3372584/samba/selftest/filter-subunit
--fail-on-empty --prefix="samba4.ldap.sort.python(ad_dc_ntvfs)."
--suffix="(ad_dc_ntvfs)"
expanded command: python
/memdisk/autobuild/fl/b3372584/samba/source4/dsdb/tests/python/sort.py
localdc -U"Administrator%locDCpass1" --workgroup=SAMBADOMAIN $LOADLIST
2>&1 | /memdisk/autobuild/fl/b3372584/samba/selftest/filter-subunit
--fail-on-empty --prefix="samba4.ldap.sort.python(ad_dc_ntvfs)."
--suffix="(ad_dc_ntvfs)"
ERROR: Testsuite[samba4.ldap.sort.python(ad_dc_ntvfs)(ad_dc_ntvfs)]
REASON: Exit code was 1
See
https://git.samba.org/autobuild.flakey.sn-devel-104/2016-03-11-1019/samba.stdout
I've attached all patches including the ldap.notification change.
Please have a look.
Can we also get someone else to review and push these?
Thanks!
metze
-------------- next part --------------
From 4efdb40c4b76bc24d7fed7edb8de23466fb3c717 Mon Sep 17 00:00:00 2001
From: Stefan Metzmacher <metze at samba.org>
Date: Fri, 11 Mar 2016 10:16:27 +0100
Subject: [PATCH 1/3] s4:dsdb/test/notification: make test_invalid_filter more
resilient against ordering races
We saw a lot of flapping tests with:
[1793(11038)/1892 at 1h55m26s]
samba4.ldap.notification.python(ad_dc_ntvfs)(ad_dc_ntvfs)
UNEXPECTED(failure):
samba4.ldap.notification.python(ad_dc_ntvfs).__main__.LDAPNotificationTest.test_max_search(ad_dc_ntvfs)
REASON: Exception: Exception: Traceback (most recent call last):
File
"/memdisk/autobuild/fl/b1782183/samba/source4/dsdb/tests/python/notification.py",
line 181, in test_max_search
self.assertEquals(num, ERR_TIME_LIMIT_EXCEEDED)
AssertionError: 11 != 3
Signed-off-by: Stefan Metzmacher <metze at samba.org>
---
source4/dsdb/tests/python/notification.py | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)
diff --git a/source4/dsdb/tests/python/notification.py b/source4/dsdb/tests/python/notification.py
index d799c91..a6dadfa 100755
--- a/source4/dsdb/tests/python/notification.py
+++ b/source4/dsdb/tests/python/notification.py
@@ -168,6 +168,8 @@ delete: otherLoginWorkstations
attrs=["name"],
controls=["notification:1"],
timeout=1)
+ num_admin_limit = 0
+ num_time_limit = 0
for i in xrange(0, max_notifications + 1):
try:
for msg in notifies[i]:
@@ -175,10 +177,15 @@ delete: otherLoginWorkstations
res = notifies[i].result()
self.fail()
except LdbError, (num, _):
- if i >= max_notifications:
- self.assertEquals(num, ERR_ADMIN_LIMIT_EXCEEDED)
- else:
- self.assertEquals(num, ERR_TIME_LIMIT_EXCEEDED)
+ if num == ERR_ADMIN_LIMIT_EXCEEDED:
+ num_admin_limit += 1
+ continue
+ if num == ERR_TIME_LIMIT_EXCEEDED:
+ num_time_limit += 1
+ continue
+ raise
+ self.assertEqual(num_admin_limit, 1)
+ self.assertEqual(num_time_limit, max_notifications)
def test_invalid_filter(self):
"""Testing invalid filters for notifications"""
--
1.9.1
From 1741aac7f4b426dadc99bc307825f03abb092904 Mon Sep 17 00:00:00 2001
From: Stefan Metzmacher <metze at samba.org>
Date: Fri, 11 Mar 2016 10:39:13 +0100
Subject: [PATCH 2/3] s4:dsdb/test/sort: avoid 'from collections import
Counter'
This is only available in python 2.7 and >= 3.1
This should fix make test with python 2.6.
Signed-off-by: Stefan Metzmacher <metze at samba.org>
---
source4/dsdb/tests/python/sort.py | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/source4/dsdb/tests/python/sort.py b/source4/dsdb/tests/python/sort.py
index c4d2c44..b7b9d83 100644
--- a/source4/dsdb/tests/python/sort.py
+++ b/source4/dsdb/tests/python/sort.py
@@ -5,7 +5,6 @@ from unicodedata import normalize
import locale
locale.setlocale(locale.LC_ALL, ('en_US', 'UTF-8'))
-from collections import Counter
import optparse
import sys
import os
@@ -191,7 +190,13 @@ class BaseSortTests(samba.tests.TestCase):
self.expected_results[k] = (fixed, list(reversed(fixed)))
for k in ('streetAddress', 'postalAddress'):
if k in self.expected_results:
- c = Counter([u[k] for u in self.users])
+ c = {}
+ for u in self.users:
+ x = u[k]
+ if x in c:
+ c[x] += 1
+ continue
+ c[x] = 1
fixed = []
for x in FIENDISH_TESTS:
fixed += [norm(x)] * c[x]
--
1.9.1
From 1a2612b5fb330b8c84ca78fb3282074ec4fd4076 Mon Sep 17 00:00:00 2001
From: Stefan Metzmacher <metze at samba.org>
Date: Fri, 11 Mar 2016 10:49:21 +0100
Subject: [PATCH 3/3] selftest: mark
samba4.winbind.struct.domain_info.ad_member as flapping
See https://lists.samba.org/archive/samba-technical/2016-March/112861.html
found 517 lines matching '^UNEXPECTED' in 641 files matching 'samba.stdout$'
175 UNEXPECTED(failure): samba4.winbind.struct.domain_info(ad_member:local)
19 UNEXPECTED(failure): samba4.winbind.struct.domain_info(s3member:local)
12 UNEXPECTED(failure): samba4.rpc.backupkey with seal.backupkey.server_wrap_encrypt_decrypt_wrong_key(ad_dc_ntvfs)
12 UNEXPECTED(failure): samba4.drs.delete_object.python(promoted_dc).delete_object.DrsDeleteObjectTestCase.test_ReplicateDeletedObject1(promoted_dc)
12 UNEXPECTED(failure): samba4.rpc.backupkey with seal.backupkey.server_wrap_decrypt_wrong_r2(ad_dc_ntvfs)
11 UNEXPECTED(failure): samba4.ldap.notification.python(ad_dc_ntvfs).__main__.LDAPNotificationTest.test_max_search(ad_dc_ntvfs)
We'll see if we also need to add
samba4.winbind.struct.domain_info.s3member
before we're able to identify and fix the problem.
Signed-off-by: Stefan Metzmacher <metze at samba.org>
---
selftest/flapping | 1 +
1 file changed, 1 insertion(+)
diff --git a/selftest/flapping b/selftest/flapping
index 81629f9..2f0ef55 100644
--- a/selftest/flapping
+++ b/selftest/flapping
@@ -29,3 +29,4 @@
^samba4.drs.delete_object.python # flakey on sn-devel
^samba4.blackbox.samba_tool_demote # flakey on sn-devel
^samba4.smb2.create.mkdir-dup\(ad_dc_ntvfs\) # This test (for bug 11486) involves a race, not always protected against in the NTVFS file server
+^samba4.winbind.struct.domain_info.ad_member # flakey on sn-devel-104 and sn-devel-144
--
1.9.1
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20160311/18a20450/signature.sig>
More information about the samba-technical
mailing list