Home

Continuous Testing: TDD Turned Up To 12

Mon, 2007-12-17 06:22 — Ben Rady
keywords:
room: Sheraton Hall C, LC — time: Tuesday 10:45-12:15
Average Rating: -

Continuous Testing (CT) is a developer practice that involves automatically running tests after every change, even so much as a single statement. It gives you instant feedback about the semantic correctness of your code, just as modern IDE’s give you instant feedback about syntax errors. CT has a profound impact on the way we use TDD. This session will cover the history, theory, practice, and daily application of CT to real-world projects.

Process/Mechanics

First, I’ll cover the history and basics of Continuous Testing, discuss it’s application on different platforms, and our ongoing experiences with CT at Semantra. Then I’ll show a demo of the practice using various languages and some open source continuous testing tools. Finally, we’ll wrap up with a discussion of where this practice might be headed.

Outline

What is Continuous Testing? (2-4 slides)

  • History
  • Benefits
  • Survey of available Open Source tools

Continuous Testing with statically-typed languages(7 - 10 slides)

  • Using dependency analysis to find a minimal tests
  • A stochastic examination of the effectiveness of test dependency analysis (see below)
  • Using CT as a method for discovering unnecessary coupling

CT for dynamically-typed language (7 - 10 slides)

  • Using CT as a substitute for code completion tools
  • Inline Commands: Controlling your development using temporary code
  • Compiling vs. Interpreting
  • Scaling to larger projects

CT @ Semantra (3 - 5 slides)

  • Realized Benefits
  • Pitfalls/Drawbacks
  • Impacts on architecture

Demo (20-30 minutes)

  • Infinitest-java
  • Zen-Test
  • Infinitest-groovy
  • Infinitest-py
  • A backdoor to CT with NUnit

Looking to the future (4-8 slides)

  • Is speed is the only relevant category for tests, or one of many factors? What if we:
    • Run the fast tests first?
    • Provide immediate feedback when tests slow down?
  • How else can we prioritize test runs?
    • Distance from last change?
    • User configurable?
  • Should “bad” tests be failing tests? What if we:
    • Fail on uncovered code
    • Fail on slow tests
  • What other opportunities does CT open?
    • Parallelization of tests across multiple cores?
    • Automatically integrate on every green bar?

Q&A (20 minutes)

A Stochastic Examination…

I’m going to set up a small experiment where:

  1. I’ll use Jumble to randomly introduce a change in a target project with a fair amount of dependency injection (probably JMock)
  2. Run all the tests to see if the change introduces any errors
  3. Run Infinitest to see if it detects the same errors
  4. Revert the change, and try again

After running this a few thousand times, I should have some solid statistical data on how often Infinitest-java finds the truly minimal set of tests. I’ll present the results of this, along with the technique, as part of the presentation.