Mutation Testing

Overview

Mutation testing is a white-box technique to help improve code coverage. In a nutshell, it is about injecting bugs into your own code and seeing if they get caught.

More specifically, it is:

1. Changing production code and its various operators. Such changes are called "mutants". Examples include:

   1. Replacing < with <=

   2. Flipping == to !=

   3. Replacing return calculationResult; with return 0;

2. Running existing tests.

3. Ensuring that at least one test failed (thus caught the change).

   1. A failing test means the introduced mutant was "killed".

   2. If all tests pass, then the mutant "survived", and it means the test suite has gaps that should be addressed.

Mutation testing is a response to some inadequate coverage types, such as Line Coverage.

Even Branch Coverage is insufficient when the tests or test data are poor, for example:

• Boundary values are not used

• Compound conditions (such as (a && b || (c && d)) ) are weakly tested

• Unless the stricter MC/DC coverage is applied, there is no guarantee that the independent influence of each condition is verified

• Missing assertions in tests (test is green, coverage exists, but nothing is really checked)

To summarize:

Coverage Practice	Added Value from Mutation Testing
Line coverage	Very high
Branch / Decision coverage	Moderate to high
Branch + MC/DC coverage	Moderate to low

Popular tools

Some of the currently popular tools. The list may change over time.

• JavaScript / TypeScript: Stryker, Mutant

• Python: mutmut, MutPy

• Java: PIT (Pitest)

• C# / .NET: Stryker.NET

• Go: Go-Mutesting