subunit - useful tools

Jelmer Vernooij jelmer at samba.org
Thu Sep 23 01:20:47 MDT 2010


You might've seen references to "subunit" while working with the Samba
testsuite ("make test" in both Samba 3 and Samba 4). subunit is a
streaming protocol used to report test results that is aimed at being
simple to generate and parse as well as being human readable. 

A very simple subunit stream might look like this:

test: samba4.tests.util.strlist.check_list_make
creating list...
list created!
success: samba4.tests.util.strlist.check_list_make
test: samba4.tests.util.strlist.check_list_make_shell
creating list...
xfail: samba4.tests.util.strlist.check_list_make_shell [
returned NT_STATUS_NOT_IMPLEMENTED
]

For those that are familiar with the TAP protocol used by Perl, it is
similar to that, although it has a couple of features that TAP does not
have.

Subunit is particularly useful for projects that use multiple
programming languages as it allows a single tool to be used for test
visualization or analysis rather than one per language, as long as there
is a test reporter for each programming language. 

selftest.pl has been using subunit internally since its creation couple
of years ago. Most other test tools can also report subunit, in
particular our python tests, blackbox tests, perl tests (using
tap2subunit) and smbtorture.

"make test" never displays raw subunit results, it always formats them
using our format-subunit script. Samba 4's "make test" stores the raw
subunit output in st/subunit.

A couple of people here at SDC have asked me about the tools I use to
display and analyse test results so I thought I'd sum them up in a mail
to the mailing list. They are:

subunit
=======

The subunit project contains a bunch of convenience tools for working
with subunit. Other than libraries for parsing/generating subunit for
several languages it contains tools for manipulating and analysing
subunit streams, including:

- subunit-ls: List all tests in a subunit stream, optionally including
their run times (I used this for the test summary I sent earlier)
- tap2subunit: convert a TAP stream to a Subunit stream
- subunit-stats: Print statistics for a subunit stream (how many
successful tests, failed tests, skipped tests, etc)
- subunit-filter: E.g. remove test result or output from a stream
- subunit-diff: Compare two subunit streams and see what tests have
started failing or are no longer failing
- subunit2pyunit: Format a subunit stream using Python's standard unit
test test result formatter

We're including the subunit tree in the Samba git tree at lib/subunit.

tribunal
========

Tribunal is a GTK+ viewer for subunit streams. It allows for easy
browsing of test results. Tribunal is still a bit rough around the
edges, although it should already be useful. 

Tribunal lives at https://launchpad.net/tribunal

Example usage:

$ make test
$ tribunal-subunit st/subunit

Homepage: https://launchpad.net/tribunal

testrepository
==============

Test Repository provides a database of test results which fits into
developers work flow and keeps track of useful information like what
tests are failing, or which failures have the same backtrace.

In particular Test Repository can re-run only the tests that failed in
the previous test run:

$ testr init
# Run the full testsuite (1 hour goes by)
$ testr run
# Run those tests from the testsuite that failed in the previous run 
# (this would be a lot shorter usually, depending on how many tests were
# failing)
$ testr run --failing

testrepository is also still in its early days, but can potentially be
very useful, e.g. when comparing old test runs on the buildfarm.

Cheers,

Jelmer
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.samba.org/pipermail/samba-technical/attachments/20100923/f7a2e5fd/attachment.pgp>


More information about the samba-technical mailing list