Main

May 03, 2006

Software Testing Benchmarks

There really aren't industry benchmarks for testing. It varies so wildly that it is useless. There is so much variation in individual IT orgs processes that no numbers make sense. I've done ignorant research in this area and have not been happy with anything out there. You essentially have to benchmark your own organization and then work to improve it. However, you are so dependent on the quality of the requirements artifacts and code that again even within your own group the variation can be incredible.

There are certain situations where benchmarking is possible. For example, if the organization follows the Use Case methodology of requirements management, then one might be able to assume that benchmarks could be used. However, again the variation in requirements detail can change everything. If you have an use case with 20 steps through the basic flow, you may be able to assume there will be 20 test steps in your test script. Yet, that use case could be documented at a high level that leaves much of the application steps out. In this case, that same 20 step use case could require 40 steps in the test script and hours of analysis on the application design to understand the user interaction.

Usually when clients bring up benchmarks they have concerns around productivity. There are other ways to approach this that can improve on that problem. The first is to assess the whole SDLC and find out what is slowing up the testing process. With respect to % of defects by lifecycle, that again can vary widely depending on the applications, the test environment and the testing done in each phase (thus the reason for no easy apples to apples industry comparisons).

Test Case Point Analysis
Using Test Maturity Models and Industry Standards for Test Process Improvement

Benchmarks are a joke in software testing though. There are ways to show the problem isn't within the testing team though (which is 99% of the case because it's usually rare that the tester truly has productivity issues and is being lazy).