[PATCH] Improve PSO test flappiness

Tim Beale timbeale at catalyst.net.nz
Mon Jun 18 20:54:25 UTC 2018


Here are a couple of minor changes that hopefully might make the
PSO/password_lockout tests a bit more robust. They basically relax the
timing window that the tests use slightly.

It's hard to know for certain because the tests always pass when run
locally, and only fail occasionally during a cloud CI run.

CI passed, results here:

Patch attached, or also here:


-------------- next part --------------
From bfc4d3c2e44accd308e5d4176dd6fab683575f96 Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Mon, 18 Jun 2018 09:03:40 +1200
Subject: [PATCH 1/2] tests: Increase minPwdAge used for PSO tests

The PSO minPwdAge test was using a 1 second timeout. While this seemed
to work fine most of the time, we did see a rackspace failure that was
presumably due to the test taking longer than 1-second to execute
(which resulted in the password not being correctly rejected).

This patch increases the minPwdAge used, to try to avoid this problem

Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
 source4/dsdb/tests/python/password_settings.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/source4/dsdb/tests/python/password_settings.py b/source4/dsdb/tests/python/password_settings.py
index d072a8f..12779a0 100644
--- a/source4/dsdb/tests/python/password_settings.py
+++ b/source4/dsdb/tests/python/password_settings.py
@@ -451,7 +451,7 @@ class PasswordSettingsTestCase(PasswordTestCase):
     def test_pso_min_age(self):
         """Tests that a PSO's min-age is enforced"""
         pso = PasswordSettings("min-age-PSO", self.ldb, password_len=10,
-                               password_age_min=1, complexity=False)
+                               password_age_min=2, complexity=False)
         # create a user and apply the PSO
@@ -462,8 +462,8 @@ class PasswordSettingsTestCase(PasswordTestCase):
         # changing the password immediately should fail, even if password is valid
         valid_password = "min-age-passwd"
         self.assert_password_invalid(user, valid_password)
-        # then trying the same password later (min-age=1sec) should succeed
-        time.sleep(1.5)
+        # then trying the same password later should succeed
+        time.sleep(pso.password_age_min + 0.5)
         self.assert_password_valid(user, valid_password)
     def test_pso_max_age(self):

From ab6f6d2fe6601bd02dc17e49c6cccd51b8208b73 Mon Sep 17 00:00:00 2001
From: Tim Beale <timbeale at catalyst.net.nz>
Date: Mon, 18 Jun 2018 09:52:57 +1200
Subject: [PATCH 2/2] tests: Increase PSO test timeouts to minimise failures

When PSOs exist in the DB, there is some extra overhead involved in user
logins (an extra expand-nested-groups operation for every user login).

Currently password_lockout tests are quite query-intensive - each call
to _check_account() does ~6 RPC operations/LDB searches (plus sleeps for
20 millisecs). Plus the actual user login attempt being tested. It looks
like the current test needs to do 3 login attempts/_check_account()
calls within a 2-second window. While the PSO test cases usually work
OK, sometimes they fail (presumably they take slightly longer and fall
outside this 2-second window). Presumably this is due to the cloud
instance's CPU being slightly more loaded when the test is run.

Long-term the plan is to refactor the user login so that the extra
expand-nested-groups operation is unnecessary for PSOs. In the
short-term, increase the window the test uses from 2 seconds to 3

Signed-off-by: Tim Beale <timbeale at catalyst.net.nz>
 source4/dsdb/tests/python/password_lockout.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/source4/dsdb/tests/python/password_lockout.py b/source4/dsdb/tests/python/password_lockout.py
index 26decf6..ec6cf13 100755
--- a/source4/dsdb/tests/python/password_lockout.py
+++ b/source4/dsdb/tests/python/password_lockout.py
@@ -618,9 +618,13 @@ userPassword: thatsAcomplPASS2XYZ
     def use_pso_lockout_settings(self, creds):
         # create a PSO with the lockout settings the test cases normally expect
         pso = PasswordSettings("lockout-PSO", self.ldb, lockout_attempts=3,
-                               lockout_duration=2)
+                               lockout_duration=3)
         self.addCleanup(self.ldb.delete, pso.dn)
+        # the test cases should sleep() for the PSO's lockoutDuration/obsvWindow
+        self.account_lockout_duration = 3
+        self.lockout_observation_window = 3
         userdn = "cn=%s,cn=users,%s" % (creds.get_username(), self.base_dn)

More information about the samba-technical mailing list