program selection system

Exploratory Test Driven Development (ETDD) is a variation of the Test Driven Development methodology. Instead of writing closed unit tests, the user writes Parameterized Unit Tests (PUTs) and lets an automated tool generate parameter values to cover all reachable statements. In this talk, we present ETDD for .net using Pex [1], an automated exploratory testing tool. Saff et al. [2] also describe this process as Theory Based Testing.

Automated Exploratory Testing

Exploratory Testing (ET) [3,4] is test design and test execution at the same time. As testing progresses, the tester learns more and more about the behavior of the code. Together with experience and creativity he can craft more and better tests.

Pex uses Dynamic Symbolic Execution, a technique that works in a way similar to ET. The tool executes the code multiple times and learns about the program behavior by monitoring the control and data flow. After each run, it picks a branch that was not covered previously, builds a constraint system (i.e. predicate over the test inputs) to reach that branch, then uses a constraint solver to come up with new test inputs, if any. The test is executed again with the new inputs, and this process repeats. On each run, Pex might discover new code and dig deeper into the implementation. We also refer to this process as Automated Exploratory Testing.

AET is exposed to the user through Parameterized Unit Tests. From a PUT authored by the user, Pex will generate a number of closed unit tests that invoke the PUT:

// add element to empty list, check count is equal to one
void AddTest(int capacity) { 
    List l = new List(capacity);
    l.Add(null); // the underlying array might be resized here
   Assert(l.Count  ==  1);
 }
// generated unit tests
void AddTestMinusOne() { this.AddTest(-1); }  // negative test; capacity cannot be negative
void AddTestZero() { this.AddTest(0); }
void AddTestOne() { this.AddTest(1); }

From Red, Green, Refactor to Red,Yellow, Green, Refactor

The TDD development flow is usually described as a short feedback loop where the developer bounces from the tests to the code and vice-versa. The step are usually 1) write a unit test, 2) watch the test fail, 3) make a minimal code change, 4) watch the test pass, 5) go to (1) (skipping the refactoring part).

ETDD shortens this loop by providing a shortcut between step (5) and (2), i.e. shortens the amount of test to write: 1) write a parameterized unit tests, 2) generate unit tests, watch a least one failing, 3) make a minimal code change, 4) watch the failing test pass, 5) go to (2) until there are no more failures, 6) go to (1). In step (2) Pex will generate more and more passing tests, and may also provide the next failing test; this is the yellow state where we have new passing tests and failing tests.

By reducing the number of tests to write by hand, the ETDD practitioner ends up spending more time writing the production code.

It’s still about Design

Although ETDD brings significant changes to TDD, it still retains Design aspect of it. When a developer writes a PUT, he designs the API and the expected behavior as the code, just as much as with a regular unit test. The difference is that, instead of focusing on a particular example with concrete values (i.e. magic numbers), PUTs let the developer express particular scenarios, without the need to invent irrelevant values. In that sense, ETDD is even more about Design than TDD.

Should I drop TDD? NO!

Developers do not have to drop TDD when doing ETDD; in some case, it’s easier to start with unit tests to get the ideas down. But at some point, one can refactor the unit test into a parameterized unit test and take advantage of the automated exploratory testing engine.

Links

[1] Microsoft Pex Home Page
[2] “Theories in Practice: Easy-to-Write Specifications that Catch Bugs”, David Saff, Marat Boshernitsan, Michael D. Ernst.
[3] “What is Exploratory Testing?”, James Bach
[4] “Exploratory Testing”, WikiPedia