GitLab merge request 200 DNS related failure

[Jeremy Allison]

On Tue, Jan 22, 2019 at 07:38:56AM +0200, Isaac Boukris wrote:
> Thanks Jeremy.
> 
> On Tue, Jan 22, 2019 at 12:59 AM Jeremy Allison <jra at samba.org> wrote:
> >
> > On Mon, Jan 21, 2019 at 11:44:50PM +0200, Isaac Boukris wrote:
> >
> > > commit 4b4ae005b2029d55ad95809a535d13eb5ff51174
> > > Author: Volker Lendecke <vl at samba.org>
> > > Date:   Wed Jan 3 13:26:54 2018 +0100
> > >
> > >     addns: Async ads_dns_lookup_srv
> >
> > Yeah this isn't the problem. It's intermittent
> > but only on gitlab-ci, on no other platform (doesn't
> > reproduce locally or on sn-devel).
> >
> > It's annoying but I believe timing related on gitlab-ci.
> > I've tried to debug this but haven't had any luck
> > so far.
> 
> I should point out that for me it does reproduce locally in a
> consistent manner, both on master and on 4b4ae005 but not on
> 4b4ae005~1. OTOH, it does seem timing related, since it doesn't fail
> if I just add '--picky-developer' to the above configure. So I guess
> that commit just impacts on the timing.

I think Tim has just nailed it from the timing
standpoint. It's not the ads_dns_lookup_srv()
code, but either startup time / response time
from the python dns server on a loaded system.

This is something we can fix by adding retries
on the DNS lookups I think. We should have those
anyway in case the join.py code is run against
a slow to respond/flakey DNS server.