Testing and Samba 4.0 (was: Re: Almsot 50% code coverage!)

[Scott Lovenberg]

On Fri, Sep 7, 2012 at 12:17 AM, Andrew Bartlett <abartlet at samba.org> wrote:

> On Thu, 2012-09-06 at 23:14 -0400, Scott Lovenberg wrote:
> > On Thu, Sep 6, 2012 at 8:31 PM, Andrew Bartlett <abartlet at samba.org>
> wrote:
> > >
> > > Ever wondered what goes on in the 90mins that an autobuild takes?
> > >
> > > One of the things that happens is that we pass over almost 50% of the
> > > code (by line):
> > >
> > > https://build.samba.org/lcov/data/coverage/samba_4_0_test/
> > >
> > > This is a particularly impressive result given the size of the
> codebase,
> > > and how many different features we implement.
> > >
> > >
> >
> > That is impressive.
> > I was debating unit testing yesterday on IRC, so I'm curious.  When
> > considering real world results, how effective do you think that
> > testing has been for Samba?  I'm interested in the results for a very
> > large, public code base.
>
> (I'm speaking here from the perspective of the AD DC effort, as I know
> that best).
>
> Testing (unit, but mostly integration) testing has been the absolute
> keystone to the AD DC effort.  Because essentially the Samba4 project
> forked and rebooted Samba, it was able to start with a very, very strong
> emphasis on testing.  The testing was originally to discover the
> undocumented details of the protocol, but of course we applied the same
> tests to our own code.  The meme was 'untested code is broken code', and
> you will see in a number of Samba4 status reports proudly including
> earlier lcov results.
>
> The selftest system grew up rapidly in those early days, and in 2007 the
> shell script driving it was reimplemented in perl.  (Jelmer is currently
> trying to re-implement the driver in python).  The subunit protocol is
> used to communicate test results, and Jelmer did a lot of work to have
> us use that.
>
> As the number of contributors to Samba4 waxed and waned, the selftest
> system and the meme of test-driven development has allowed new
> developers to come up to speed writing tests as they wrote code, and
> giving the rest of us some confidence that they were not breaking some
> other code while improving things.
>
> However, the crowning achievement of our test system was when autobuild
> was adopted, which I think was about 2 years ago.  The days of waiting
> for the build farm results to see how badly we had regressed are over,
> and it is no longer a challenge of chasing back in time to find who
> broke the build, or trying to fix someone else's code.  Instead, code
> only gets into the tree when it passes that battery of tests.
>
> The timing was fortuitous, as 18 months ago, the team I was a part of at
> Cisco doing Samba 4.0 AD work disbanded, and the available hands on
> Samba4 changed (as other companies took up the slack for a time) and
> then plummeted.  With autobuild in place, it remained working
> consistently despite all the ongoing changes, and as we made the
> transition to using the smbd file server.
>
> autobuild and our tests made it possible to continue development,
> particularly as resources changed.  That we can be confident in the
> quality of the AD DC in the 4.0 release I credit to that strong history
> of test driven development, and the enforcement of autobuild.
>
> Without  automation like this, with myself as the only 'full time' (I'm
> actually part time) developer focussed on the AD effort, I know I would
> be spending my whole time fire-fighting (because that is what I did a
> lot of before we had autobuild).
>
> I'm sure this is longer than you expected, but I hope this helps give
> you a feeling for the background here.
>
> Andrew Bartlett
> --
> Andrew Bartlett                                http://samba.org/~abartlet/
> Authentication Developer, Samba Team           http://samba.org
>
>
>
That was a great read and I'm sure I'm not the only one that it was
beneficial for.  Thank you for taking the time to share that on the list. :)

-- 
Peace and Blessings,
-Scott.