autobuild failure on sn-devel-144 for task ctdb during test

[Amitay Isaacs]

On Tue, Aug 9, 2016 at 6:07 PM, Stefan Metzmacher <metze at samba.org> wrote:

> Hi Amitay and Martin,
>
> I thought the tests/simple/54_transaction_loop_recovery.sh test
> would be stable again in master, but an autobuild with
>
> https://git.samba.org/?p=metze/samba/wip.git;a=shortlog;h=
> 720df7b3a91215a5a8e5670c5e2587cc9795c1de
>
> Failed again :-(
>
> In ctdb.stderr I got:
>
> >> NODE: /memdisk/metze/W/b381815/ctdb/ctdb/tests/var/sock.0 <<
>
> >> NODE: /memdisk/metze/W/b381815/ctdb/ctdb/tests/var/sock.1 <<
>
> >> NODE: /memdisk/metze/W/b381815/ctdb/ctdb/tests/var/sock.2 <<
> tests/scripts/integration.bash: line 572: 847085 Terminated
> recovery_loop > /dev/null
> make: *** [autotest] Error 1
>
> In ctdb.stdout:
>
> ALL OK: tests/simple/53_transaction_loop.sh
> ==========================================================================
> TEST PASSED: tests/simple/53_transaction_loop.sh (duration: 32s)
> ==========================================================================
> --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--
> Running test tests/simple/54_transaction_loop_recovery.sh (21:21:45)
> --==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--==--
> Cluster is HEALTHY
> Starting recovery loop
> Running transaction_loop on all 3 nodes.
> Failed to execute
> "/memdisk/metze/W/b381815/ctdb/ctdb/tests/scripts/test_wrap
> transaction_loop         -n 3 -t 30" on node(s) "all"
> Waiting for cluster
> transaction loop test failed
> Transaction[2]:      0   7751   7135
> Transaction[1]:      0   7751   7109
> *** TEST COMPLETED (RC=1) AT 2016-08-08 21:22:19, CLEANING UP...
> Restarting CTDB (scheduled)...
> Attempting to politely shutdown daemons...
> Sleeping for a while...
> =1|.|
> Starting 3 ctdb daemons...
> Node 2 will have no public IPs.
> Waiting for cluster to become ready...
> <120|...........|11|
> OK
> Setting RerecoveryTimeout to 1
> Forcing a recovery...
> =2|..|
> Doing a sync...
> ctdb is ready
> ==========================================================================
> TEST FAILED: tests/simple/54_transaction_loop_recovery.sh (status 1)
> (duration: 59s)
> ==========================================================================
>
> Can you please have a look?
>
> Thanks!
> metze
>
>
I tried running several test autobuilds to reproduce this problem, but I
could not.

I have a patch that adds extra logging to the test, so it will be easy to
debug failure.

Amitay.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-ctdb-tests-Log-errors-if-the-test-fails.patch
Type: application/text
Size: 1368 bytes
Desc: not available
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20160810/7027134b/0001-ctdb-tests-Log-errors-if-the-test-fails.bin>