program selection system

Despite the advances in both developer and customer automated testing, there remains a great deal of anecdotal evidence and testing thought leadership that suggests degrees of manual testing is still required for software-intensive projects, especially at the traditional, higher test levels such as user acceptance testing. We have come to believe that this doesn’t mean agile techniques do not stay in the picture for the duration of manual testing. We have found that using Scrum-like management techniques are highly effective in planning, tracking, leading and organizing manual testing efforts, especially in the context of experienced testers that know both the system under test and the business domain that it supports.

Our classical example is an enterprise system upgrade that affected over 4000 business users and involved upgrading business-critical processes such as finance, human resources (less payroll), and logistics, among other business functions. For the manual testing phases that were on the critical path, each testing sub-team was encouraged to devise a testing backlog that consisted of ‘test targets’ - that is, a list of items that were to be tested. We have found that focusing on ‘test targets’ instead of ‘test cases’ improves communication because we can get feedback quicker. Part of the cost of testing is finalizing the test cases, test scripts, and test data - however, if the targets of the test can be listed first, discussed, posted publicly, all without the effort to craft the scripts or test data, then we have adequately encapsulated a proxy for the ‘size’ of the test effort, very quickly. This proved valuable in validating the project schedule.

Each testing backlog was prioritized and ordered by the sub-teams and then stored in the project’s repository. When the testing began, the teams were asked to update their testing backlog directly in the repository so that the project’s test coordinator could use some worksheet-linking magic to create a combined burn-down chart that illustrated actual progress against expected progress. One flaw in the process that we acknowledged but didn’t bother to do anything about was that each of the test targets was assumed to be of the same complexity and require approximately the same effort to test - in reality, we knew this wasn’t true. Things tended to balance out in the end anyway, and the backlogs/burn-down combination proved to be highly effective in communicating testing progress and potential trouble spots. Although we tried daily scrums to support the testers, as the pressure increased on the project, the meetings were increasingly influenced by the project managers and they became ‘estimated time to complete’ meetings and not very good scrums. In the future we propose to exclude project managers from attending the daily testing scrums. In this particular context, the sub-team leaders were already co-located and were communicating effectively without the scrums, having worked for many years together and having already been through a similar upgrade together several years earlier.

To accommodate the audit and control requirements associated with the project, the testing backlogs were designed to include links to evidence that the tests were completed and that the expected results were achieved. At times this was an internal system document identifier, and at other times this was a worksheet that contained a log of the test run, complete with screen shots and analysis that supported the test result. The test environment (application and data) was backed up and stored at the conclusion of the test effort in order to support the requirement of reproducing test results if requested by internal or external auditors. Some of the test targets were considered critical enough that formal test scripts were devised.

In summary, we have devised a test management scheme that is based on agile principles and methods and that is highly effective for manual test efforts such as user acceptance testing. There are benefits from identifying the targets of the testing, prioritizing the targets, and communicating them to other members of the team. The testing planning effort was ultimately more public than it would have been without the testing backlogs. The idea of a ‘test target’ that could be identified, prioritized, and communicated without a huge expenditure of effort contributed to that effectiveness. During testing, the backlog and associated burn-down charts were extremely useful in tracking the test effort non-invasively, that is, without asking awkward estimated-time-to-complete questions. Further, we observed that the same testing backlogs supported both scripted and non-scripted (exploratory) testing methods - and that they proved to provide just enough context for making the scripted/non-scripted decision early on in the project.