Testing and Samba 4.0 (was: Re: Almsot 50% code coverage!)

[Andrew Bartlett]

On Thu, 2012-09-06 at 23:14 -0400, Scott Lovenberg wrote:
> On Thu, Sep 6, 2012 at 8:31 PM, Andrew Bartlett <abartlet at samba.org> wrote:
> >
> > Ever wondered what goes on in the 90mins that an autobuild takes?
> >
> > One of the things that happens is that we pass over almost 50% of the
> > code (by line):
> >
> > https://build.samba.org/lcov/data/coverage/samba_4_0_test/
> >
> > This is a particularly impressive result given the size of the codebase,
> > and how many different features we implement.
> >
> >
> 
> That is impressive.
> I was debating unit testing yesterday on IRC, so I'm curious.  When
> considering real world results, how effective do you think that
> testing has been for Samba?  I'm interested in the results for a very
> large, public code base.

(I'm speaking here from the perspective of the AD DC effort, as I know
that best). 

Testing (unit, but mostly integration) testing has been the absolute
keystone to the AD DC effort.  Because essentially the Samba4 project
forked and rebooted Samba, it was able to start with a very, very strong
emphasis on testing.  The testing was originally to discover the
undocumented details of the protocol, but of course we applied the same
tests to our own code.  The meme was 'untested code is broken code', and
you will see in a number of Samba4 status reports proudly including
earlier lcov results. 

The selftest system grew up rapidly in those early days, and in 2007 the
shell script driving it was reimplemented in perl.  (Jelmer is currently
trying to re-implement the driver in python).  The subunit protocol is
used to communicate test results, and Jelmer did a lot of work to have
us use that. 

As the number of contributors to Samba4 waxed and waned, the selftest
system and the meme of test-driven development has allowed new
developers to come up to speed writing tests as they wrote code, and
giving the rest of us some confidence that they were not breaking some
other code while improving things.  

However, the crowning achievement of our test system was when autobuild
was adopted, which I think was about 2 years ago.  The days of waiting
for the build farm results to see how badly we had regressed are over,
and it is no longer a challenge of chasing back in time to find who
broke the build, or trying to fix someone else's code.  Instead, code
only gets into the tree when it passes that battery of tests.

The timing was fortuitous, as 18 months ago, the team I was a part of at
Cisco doing Samba 4.0 AD work disbanded, and the available hands on
Samba4 changed (as other companies took up the slack for a time) and
then plummeted.  With autobuild in place, it remained working
consistently despite all the ongoing changes, and as we made the
transition to using the smbd file server.  

autobuild and our tests made it possible to continue development,
particularly as resources changed.  That we can be confident in the
quality of the AD DC in the 4.0 release I credit to that strong history
of test driven development, and the enforcement of autobuild.  

Without  automation like this, with myself as the only 'full time' (I'm
actually part time) developer focussed on the AD effort, I know I would
be spending my whole time fire-fighting (because that is what I did a
lot of before we had autobuild). 

I'm sure this is longer than you expected, but I hope this helps give
you a feeling for the background here.  

Andrew Bartlett
-- 
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org