RFI Marick
= Biggest problem:
Automated functional testing lives between two sensible testing activities. On the one side, there’s conventional TDD (unit testing). On the other side, there’s manual exploratory testing. It is probably more important to get good at these than it is to get good at automated functional testing. Once you’ve gotten good at them, what does it mean to get good at automated functional testing?
There is some value in thinking through larger-scale issues (such as workflows or system states) before diving into TDD. There is some value (but not, I think, as much as most people think) in being able to rerun larger-scale functional tests easily. That is: compared to doing exploratory testing and TDD right, the testing we’re talking about has modest value. Right now, the cost is more than modest, to the point where I question whether a lot of projects are really getting adequate ROI. I see projects pouring resources into functional testing not because they really value it but more because they know they should value it. (And because they’ve got these testers, and they have to do something with them, and they know manual testing is bad…)
This is strikingly similar to, well, the way that automated testing always worked in the pre-Agile era: often a triumph of hope over experience.
I’m betting today that the point of maximum leverage is reducing the cost of larger-scale testing. Right now, those simple workflow statements and checks are annoyingly hard to implement. Even I, staring at a workflow test, get depressed at how much work it will be to get it running, compared to all the other things I could be doing with my time.
Why does test implementation cost so much?
We are taught that Agile development is about working the code base so that arbitrary new requirements are easy to implement. We have learned one cannot accomplish that by “layering” new features onto an existing core. Instead, the core has to be continually massaged so that, at any given moment, it appears as if it were carefully designed to satisfy the features it supports. Over time, that continual massaging results in a core that invites new features because it’s positively poised to change.
What do we do when we write test support code for automated large-scale tests? We layer it on top of the existing system (either on top of the GUI or on top of some layer below the GUI). We do not work the new code into the existing core - so, in a way that ought not to surprise us, it never gets easier to add tests.
So the problem is to work the test code into the core. The way I propose to do that is to take exploratory testing more seriously: treat it as a legitimate source of user stories treated just like other user stories. For example, if an exploratory tester wants an “undo” feature for a webapp, implementing that feature will have architectural consequences (such as moving from requests-call-controller-action-methods to requests-create-command-objects).
Why drive the code with exploratory testing stories rather than functional testing stories? I’m not sure. It feels right to me for several nebulous reasons I won’t try to explain here.
= What do I hope to contribute?
A session in which we create stories for a webapp. Basically, I want to hear manual testers describe the sorts of things they wish they could do when testing their webapps.
More sessions in which a few of us take a web framework (Ramaze, I hope), and start implementing some of those stories.
= What do I hope to get?
Some momentum toward working on the framework to make it an exemplar of a test-supporting web framework.