autobuild failure due to python replication tests - why ?

[Jeremy Allison]

On Tue, Aug 02, 2016 at 01:49:34PM +1200, Andrew Bartlett wrote:
> On Mon, 2016-08-01 at 17:28 -0700, Jeremy Allison wrote:
> > 
> > Why is this happening ? It's very frustrating to
> > try and get simple code changes in and find that
> > they're stimied by unrelated issues like this.
> 
> To do understand your frustration!  We have tried a number of
> approaches and invested ourselves in a number of different proposed
> resolutions that turn our not to help.
> 
> The issue is that the underlying issue appears to be historical, not a
> regression, and these tests fail in a load-dependent manner. 
> 
> They almost never failed when tested in isolated VMs with 4 CPUs
> allocated.  When I was finishing the last big DRS replication test, I
> had 10 tests passing or failing on unrelated issues in the Catalyst
> Cloud.
> 
> Additionally, at the time that they were added, they were reliable on
> sn-devel (otherwise I wouldn't have added them), at least as much as
> the rest of the test was reliable at that time.  (I may have seen one
> failure).

I'm sure that's the case. No one adds broken tests deliberately
and I don't think you did that.

> Finally, we have addressed some of the other flapping tests in the file
> server and elsewhere in the AD DC, so now we progress to the
> replication tests.  Attached is a fix for another such flapping test. 
> 
> This brings us to the current situation: 
> 
> These replication tests and a number of CTDB tests reliably fail under
> load.  As we rushed to get 4.5 branched for example, that seems to have
> become a particular issue. 

Then the replication tests and ctdb tests NEED TO BE REMOVED NOW !

Until they are reliable under all circumstances, they are worse
than useless.

They are actively preventing development. I don't think that
is an allowable state of affairs.

If you want to keep them, you must give me a date by which
they will be fixed. Otherwise I'll propose a patch that
removes them and get it reviewed and pushed.