> For the complete documentation index, see [llms.txt](https://practical-testing.gitbook.io/home/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://practical-testing.gitbook.io/home/testing-practices/reporting-with-testing-metrics.md).

# Reporting with Testing Metrics

> "You can’t control what you can’t measure" 
>
> Tom DeMarco, author of Controlling Software Projects

### The desire for control

It is natural for humans to want to track progress in personal life, work projects, or business. The logical conclusion is to track metrics - "bad" numbers must stay low, "good" numbers must go up.

On a software project, you may have individual, process, project, and higher-level metrics. There are dozens, if not hundreds, of metrics used in testing. It can be argued that **most are poor or useless.**

{% hint style="warning" %}
Most metrics related to testing are poor, useless, or vulnerable to abuse ([Goodhart's Law](#user-content-fn-1)[^1]).
{% endhint %}

### Poor metrics and their abuses

Some places may keep track of the number of (manual) test scenarios produced per day, or automation scripts written per sprint, to measure a tester's efficiency. This has no value, because intellectual work is very different from manual labor.

* No one judges a book by the number of words it contains
* No one judges a project manager by the number of emails they send per week
* And it is well-known that developers shouldn't be judged based on the number of Lines of Code (LoC) written or Pull Requests created - quantitative metrics that are detached from quality or delivered business value.

Below is a non-exhaustive list of some of the worst testing metrics that you should probably never use. If management expresses a desire to track them, consider explaining to them why they are not a good idea.

<table><thead><tr><th width="181">Metric</th><th>Why it is a poor metric</th><th>How it can be abused or gamed</th></tr></thead><tbody><tr><td>Number of test cases written</td><td>Quantity says nothing about relevance, depth, or effectiveness</td><td>Inflate test cases by splitting trivial variations</td></tr><tr><td>Automation scripts written</td><td><p>As above. </p><p></p><p>Additionally, scripts vary hugely in value and complexity</p></td><td>Write shallow or redundant scripts</td></tr><tr><td>Test cases executed</td><td>Execution does not imply meaningful coverage</td><td>Re-run easy or low-value tests to boost numbers</td></tr><tr><td>Bug count found by testers in the "testing phase"</td><td>At odds with shift-left testing and mindset.<br><br>Isn't it more cost-effective for testers to help developers and the team to spot problems as early as possible in the SDLC?</td><td>Testers are incencitivized to find and report bugs later, rather than catching errors before they are coded and deployed to a testing environment. </td></tr><tr><td>Bug count per tester</td><td>Penalizes collaboration</td><td>Avoid helping others or focus on easy areas</td></tr><tr><td>Severe bug count per tester</td><td>As above + incentivizes testers to inflate the importance of bugs found</td><td>As above + increases arguing if "this bug is really that severe"</td></tr><tr><td>Test coverage (%)*</td><td>Coverage ≠ quality or meaningful test data or scenarios</td><td>Add superficial tests to raise coverage</td></tr><tr><td>Test execution speed</td><td>Speed alone ignores learning and investigation</td><td>Rush testing and miss important issues</td></tr><tr><td>Bugs found in production</td><td>Influenced by many factors outside testers' direct control**</td><td>When used as a punishment, leads to toxic blame games</td></tr></tbody></table>

**\*Test coverage is a vague term**. From a black-box perspective, it could mean "requirements" or "feature" coverage. This again has severe caveats:

* "*I created enough tests to achieve 100% coverage*" actually means "*coverage with whatever scenarios I could think of. I can't possibly know if I missed or misunderstood something, or whether my test data will miss a bug still. If I knew - I would've written them down*".
* 100% coverage of incomplete, inconsistent, or just wrong requirements still leads to a poor product.

\
\*\* **Defect counts are driven largely by factors outside the tester's control**, such as:

* requirements clarity and completeness
* quality of the code developers produce
* degree of technical debt already present in the system

A QA team may find few bugs because developers delivered high-quality work; **penalizing the team for this is conceptually flawed**. Conversely, a high defect count often indicates underlying issues in the earlier phases of the SDLC, rather than superior QA performance.

### Potentially better metrics

{% hint style="info" %}
Remember, no metric is completely immune to Goodhart's law or other abuse
{% endhint %}

Not tracking anything is probably another extreme. So what should we track?

First, establish a team-wide mindset:

* **Prevention** rather than detection. The earlier the better.&#x20;
* **Strong collaboration**. Joint quality ownership. No "throw it over the wall and let testers do their thing".&#x20;
* **Fast feedback loops in CI/CD pipelines**

#### White-box coverage

High code coverage is still a valid metric, **assuming the automated tests are written intelligently using various test techniques described on this resource and with quality test data**.&#x20;

<table><thead><tr><th width="175">Metric</th><th width="308">Why it's useful</th><th>How it supports quality</th></tr></thead><tbody><tr><td>Branch coverage</td><td>Ensures both true and false paths of decision points are executed. <br><br>With quality test data, it may prove every decision outcome is effective and behaves correctly at borders of equivalence partitions.</td><td>When high, greatly reduces the risk of untested conditional logic and edge cases</td></tr></tbody></table>

#### Black-box or Quality Control coverage

All of the below metrics should be treated as a learning signal, not a performance KPI.

<table><thead><tr><th width="195">Metric</th><th>Why it’s more useful</th><th>How it supports quality</th></tr></thead><tbody><tr><td>Bugs found in production over time</td><td>Focuses on impactful issues as a trend, not a snapshot in time.<br><br>This metric (without "over time") is also listed in the "Poor Metrics" table above. This is intentional. It depends on how this metric is used - to punish or to analyze and improve?</td><td>Helps gauge real-world risk escape. <br><br><strong>Important!</strong> There must be no blaming, only an honest discussion of why it happened and how it can be prevented in the future.</td></tr><tr><td>User ratings and reviews/feedback over time</td><td>100% coverage or a million test cases will not matter if the end users end up disliking the product and switching to the competition.</td><td>Ratings can be measured. Reviews may be qualitatively assessed. </td></tr><tr><td>Mean Time to Detect (MTTD)*</td><td>Measures how quickly problems are noticed after introduction</td><td>Tracks feedback loop efficiency</td></tr><tr><td>Mean Time to Repair (MTTR)</td><td>Shows how fast issues are resolved when found</td><td>Indicates team responsiveness and process health</td></tr></tbody></table>

\*MTTD raises serious questions. **You usually can’t know the exact moment a defect was introduced.** When a Business Analyst typed it into a JIRA ticket? The moment a developer typed it in code? MTTD is almost always an **approximation**, and sometimes a **proxy metric**.

To handle this in practice, teams may **define a consistent reference point**, even if it’s imperfect:

1. **Code-related defects**\
   \&#xNAN;*Introduced time* ≈ commit or merge time
2. **Requirements or design defects**\
   \&#xNAN;*Introduced time* ≈ requirement approval, story “Ready”, or sprint start (artificial, but good enough)

MTTD may still be acceptable (with caveats) if used to track **feedback speed**, not root cause timing.

Admittedly, the biggest drawback of formally following MTTD is a lot of meta-work (bureaucracy). In well-communicating teams, the overhead outweighs the value.

[^1]: States that "when a measure becomes a target, it ceases to be a good measure," meaning that people will manipulate a metric to meet the goal, causing the metric to lose its original usefulness and leading to unintended consequences, like teachers "teaching to the test" or companies focusing on short-term sales numbers over long-term customer satisfaction.