program selection system

Testing a program by comparing text files written by the current run with those from previous runs is an old idea, but one which has fallen out of fashion. This session aims to rehabilitate it as an agile technique and present a suitable tool, TextTest. We aim to show that there are applications where testing in this way has major advantages, and examine the advantages and disadvantages in a more general setting. We will include a practical demonstration showcasing Behaviour-Driven Development, but not as you know it, with TextTest.

The above paragraph is for the program: the original description follows.

Testing a program by running it and comparing its output against a “gold standard” version of that output is an old idea, but one which has fallen out of fashion in recent years. This can, however, be an extremely effective technique in some circumstances, given comprehensible output and good tool support. Developing against such tests leads to a very “behaviour-driven” development style where the tests are entirely focussed on how the program is called and how it behaves in response, both in terms of actual output and logged information.

The currently favoured approaches to testing involve calling into the system via some sort of test API, and making assertions about returned values. Acceptance test tools generally keep this model and add an abstraction layer above the API calls to aid non-technical people in writing tests. But there are situations where the text-based paradigm has major advantages over this paradigm, such as:

• Small UNIX-style command-line scripts (e.g. build scripts). The command line is their natural environment anyway and creating APIs is overkill.
• Legacy systems. Improving the logging in a system is far lower-risk to do without tests than creating a test API.
• Unusual or older languages. Code-level tools need translating for every new language, and the translations are often far worse than the original tool. A command-line tool works for every language past, present and future with no changes.

There are even generic advantages to the setup on more general systems, especially in the context of an agile process, such as:

• Doesn’t require hardcoding a “correct behaviour”. Many things are interesting to test but have no clear “objectively” correct behaviour. One of the assumptions of Agile processes is that the correct behaviour of the system will change over time, and this approach makes it easy to do this.
• Independence from internal design makes major structural refactoring easier, as it cannot break any tests
• Can use the same technique and tests for all levels of detail, from customer-relevant “external behaviour” down to developer-relevant code behaviour. Leads to a single test suite which the whole team can own.

Naturally there are disadvantages too, which will also be discussed, along with how to mitigate them:

• The “Expert Reads Output” antipattern, where people get so used to correct behaviour changes that they stop checking them properly before accepting them.
• Potential performance overhead of logging, even when disabled.

We will present these ideas along with the tool TextTest (http://www.texttest.org), which is an open source tool that tests in this way. This is a stable, feature-rich tool that has been around for over five years, and is used heavily within our organisations and a few others besides. Features include:

• regexp-based filtering of output pre comparison
• parallel testing on grids, by interfacing with grid engines like Sun Grid Engine and LSF
• advanced “mocking” capabilities, by intercepting at the command-line or network-traffic level without changing the system code
• test management and test data management
• various interpretation tools to associate textual patterns with behaviour descriptions or bugs in bug systems
• HTML reports that track behaviour of nightly test runs over time
• Built-in performance testing