autobuild failure due to python replication tests - why ?

[Jeremy Allison]

Hi Andrew,

I just tried to push a very simple cleanup path
and the autobuild failed with the following after
213 minutes:

1922(13349)/1951 at 3h1m8s] samba4.drs.repl_move.python(promoted_dc)(promoted_dc)
UNEXPECTED(error): samba4.drs.repl_move.python(promoted_dc).repl_move.DrsMoveBetweenTreeOfObjectTestCase.test_ReplicateMoveInTree3b(pr
REASON: Exception: Exception: Traceback (most recent call last):
  File "/memdisk/jra/a/b940082/samba/source4/torture/drs/python/repl_move.py", line 1838, in setUp
    self._net_drs_replicate(DC=self.dnsname_dc1, fromDC=self.dnsname_dc2, forced=True)
  File "/memdisk/jra/a/b940082/samba/source4/torture/drs/python/drs_base.py", line 119, in _net_drs_replicate
    return self.check_output(cmd_line)
  File "bin/python/samba/tests/__init__.py", line 804, in check_output
    raise BlackboxProcessError(retcode, line, p.stdout.read(), p.stderr.read())
BlackboxProcessError: Command '/memdisk/jra/a/b940082/samba/bin/samba-tool drs replicate -USAMBADOMAIN/Administrator%locDCpass1 --sync
  File "bin/python/samba/netcmd/drs.py", line 368, in run
    drs_utils.sendDsReplicaSync(server_bind, server_bind_handle, source_dsa_guid, NC, req_options)
  File "bin/python/samba/drs_utils.py", line 83, in sendDsReplicaSync
    raise drsException("DsReplicaSync failed %s" % estr)
'

Why is this happening ? It's very frustrating to
try and get simple code changes in and find that
they're stimied by unrelated issues like this.

I think we need to have some clear guidelines
regarding the tests:

1). If a make test passes on a local machine
but fails when pushed to autobuild due to an unrelated issue
then the failed test should have (1) black
mark added to it.

2). After (2) black marks (or should this be
3 or more ?) the test should automatically
be added to the flaky tests and failures
ignored.

3). If more work is done on the tests that
should fix it, it is re-enabled and the
process started again.

The unreliability of the python DC tests
is becoming a serious problem in our development
schedule, and something *MUST* be done about
this.

Thoughts anyone ?

Jeremy.