Code Coverage Types

Only CODE coverage types are covered on this page, thus basic programming knowledge is a prerequisite. Code examples are in Java, but they are simple enough to be universally understandable.

"What exactly do you mean by 80% coverage?"

Quick Summary

• Many code coverage types exist. Some are more thorough or rigorous than others:

  • Least thorough: Line, Statement, Function

  • Better: Branch / Decision

  • More thorough: Condition, Modified Condition/Decision (MC/DC)

• If you say "We have 80% code coverage", you MUST be able to elaborate or specify which type you mean if anyone asks.

• Bugs are still very likely, even with 100% code coverage of any type.

  • Example: code for unexpected input or events that was simply never written (division by zero, unsupported character, abruptly terminated connection, "file not found", etc.)

Definition and caveats

Code coverage is a metric that shows the percentage of your code that is covered by tests.

Code coverage does NOT tell you how good the tests are. Do they test partition boundaries? Do they include important special values? Etc. Code coverage does NOT, and CANNOT tell you anything about ABSENT code - the code you've never written to handle the scenarios you've never thought of.

Line and Statement Coverage

These 2 coverage types are simplest to understand, but they are also the least rigorous. They also often show identical results.

// Program v1:
// 2 lines, 2 statements
// program executes sequentially: 100% line and 100% statement coverage

count++; 
updateTotal(count);

// Program v2:
// 1 line, 2 statements (separated by a semicolon)
// program still executes sequentially: 100% line and 100% statement coverage

count++; updateTotal(count);

Line and statement coverage work well with trivial lines of code with no special logic. The two metrics start diverging when branching gets involved, and 2+ statements are written on 1 line.

// 1 line, 2 statements
// if isAdult = false, then approve() statement will never run
// in this case: 100% line coverage, 50% statement coverage

if(isAdult) approve();

If you believe the above example is contrived, consider that Python's popular list comprehensions (an alternative to loops) are one-liners that may contain a lot of logic, including branching and value transformations.

# reads as "loop over items, if item X is valid, add X*X to result list"
# 100% line coverage, 
# <100% statement coverage, depending on is_valid() implementation and value of x
result = [x * x for x in items if is_valid(x)]

Function Coverage

// Any ONE invocation with ANY input means 100% coverage for this function
int calculate(int a, int b) {
    // very long and complex computation
    // spanning 20-30 lines
}

Similar to line and statement coverage, function coverage can only serve as an indicator of large coverage gaps, i.e. something that hasn't been tested at all.

However, a function (or a method) with several parameters typically requires much more rigorous testing. See coverage types below.

Control Flow Coverage Types

There are many "Control Flow" coverage types, many with similar names and very fine-line distinctions:

• Decision / Branch Coverage (DC)

• Condition Coverage (CC)

• Condition/Decision Coverage (CDC)

• Modified Condition / Decision Coverage (MC/DC)

• Multiple Condition Coverage (MCC)

The rest of the page explains the fine differences between these.

TLDR: CDC and MC/DC are balanced, yet rigorous enough approaches for most cases.

Note that some Coverage Tools (e.g. Intellij's internal coverage counter) use them, yet still somewhat confusingly calls it "Branch".

Before diving into each type, consider these requirements:

Approve: if the applicant is an adult AND they have a job AND their monthly salary is over 1000

Reject: in all other cases

Below are the graph and the code representing such a requirement.

Requirements Graph

Requirements Code

From the graph and code, it follows:

Count	What	Comment
1	Code-level decision	if(decision){ }
2	Business outcomes (approve / reject) and thus 2 code branches
3 (N)	The decision is comprised of 3 (N) conditions	aka atomic boolean expressions, predicates
4 (N + 1)	Graph orange Leaf Nodes	Logical end of any control flow path
6 (N*2)	Graph blue branches. They also correspond to code condition outcomes.	Each decision has 2 outcomes (true/false). Here 3 decisions * 2 outcomes = 6. isAdult has 2 outcomes hasJob has 2 outcomes salary ≥ 1000 has 2 outcomes
8 (2^N)	Number of all possible condition outcomes.	Corresponds to the Full Decision Table.

Branch / Decision Coverage

Coverage criteria: Every decision in the program has all possible outcomes (code branches) at least once.

// Min. Tests needed - any combination that leads to triggering both branches
// Test 1: isAdult = true; salary = 2000
// Test 2: isAdult = true; salary = 500
if (isAdult && hasJob && salary >= 1000) {
    approve();    // Test 1 triggers Outcome / Branch 1
} else {
    reject();     // Test 2 triggers Outcome / Branch 2
}

"Branch" coverage most likely means something else in your Test Coverage tool. It definitely does in IntelliJ and other IDEs and tools by JetBrains.

Condition Coverage (CC)

Coverage criteria: Every condition in a decision in the has taken all possible outcomes at least once

// For 100% coverage, code must run with:

// isAdult        both true and false
// hasJob         both true and false
// salary >= 1000 both true and false

if (isAdult && hasJob && salary >= 1000) {
    approve();    
} else {
    reject();     
}

// That's potentially 3*2=6 tests, 
// But only 4 (N+1) tests are needed to satisfy the criteria
// From the code perspective, because control flow short-circuits out of the evaluation
// i.e. if isAdult=false -> other values don't matter anymore

From the graph perspective, consider that one test can cover many edges (graph branches).

Condition / Decision Coverage (CDC)

Coverage criteria: as the name implies, it is simply the combination of Condition + Decision coverage. In this simple example, and in many cases, Condition coverage naturally leads to Decision coverage as well.

Modified Condition / Decision Coverage (MC/DC)

Coverage criteria: Every condition in a decision has shown to independently affect decision's outcome.

In this example, while holding other conditions constant, we must demonstrate that:

• isAdult independently affects the decision

• hasJob independently affects the decision

• salary >= 1000 independently affects the decision

In this example, again, CC, CDC and MC/DC can all be satisfied with the same set of 4 tests.

MC/DC distinction becomes more apparent when conditions are masked, redundant, or logically subsumed (e.g. A && (B ||C) ).

For a more thorough explanation, see this subpage tutorial.

For a more general demonstration, see Elementary Comparison.

Multiple Condition Coverage (MCC)

Coverage criteria: Every combination of condition outcomes within a decision has been invoked at least once.

With 3 conditions, we get 2^3=8 combinations, thus 8 tests. This corresponds going through a full, non-collapsed Decision Table.

This is the most thorough, yet rarely practical approach, because the number of combinations grows exponentially. (6 conditions already leads to 2^6=64 tests).

"Branch" Coverage in Tools

Many tools may report "Branch" Coverage, but it actually means something else.

Take a moment and understand what your tool means by "Branch", so you understand the strengths and weaknesses of the underlying approach.

For example, IntelliJ IDEA (Java) doesn't look at source code conditions (1), NOR at source code conditional "branches", but at bytecode branches (2). There, it sees 3 branches, each with 2 possible outcomes, hence it counts 6 branches (3) to cover.

In practice, however, bytecode branch coverage + statement coverage should almost always correspond to at least Condition / Decision Coverage (CDC) or even MC/DC.

As such, the coverage tools for interpreted languages such as JavaScript or Python should offer a rigorous underlying coverage type.

It's best to know what exactly your tool means by "branch coverage" to know its strengths and weaknesses.

Where 100% Coverage Fails

Off-by-one errors

Control flow coverage is not concerned with test data quality.

For salary > 1000, it is enough to select random values, such as 500 and 2000, to achieve "full branch / condition coverage".

But if the requirements said "salary 1000 or more", then salary > 1000, clearly has a bug, it should be salary >= 1000.

This example demonstrates that coverage metrics are blind to the gap between source code and requirements, implicit or explicit.

Border values are better - at least 999 and 1000, though 1001 could be added for additional thoroughness.

Improve tests driven by "branch coverage" with better test data using EP & BVA

The code that never was

Internally, some function process(input1,input2,input3) might fetch a file over the network, get a value from the file, use a complex formula to process the inputs, and save the result to another file.

With 100% function, statement, and branch coverage (even MCC - the most exhaustive one), the following bugs occur:

1. 🐛 Oops, the network connection failed.

2. 🐛 Oops, the file wasn't there

3. 🐛 Oops, the file was empty or had corrupted values

4. 🐛 Oops, the value calculated by one function was 0, given to another function, and a division by zero error occurred

5. 🐛 Oops, the sum (result of a multiplication) turned out to be bigger than what an Integer variable can hold, so either rounding happened, or an exception was raised

6. 🐛 Oops, the result failed to be saved to a file (full disk, no permission, other)

7. 🐛 Oops, the result was saved too late, causing a dependent service to fail when it tried to consume it.

8. 🐛 Oops, one of the requested features was overlooked and not implemented!

All of the above and many more failures might occur because no code was written for such scenarios. No one ever thought of them!

Coverage metrics, by definition, can only evaluate what is in code, and many precautions might be missing. It could be argued that most challenging bugs occur at the integration level.

High code coverage with carefully picked test values is a good start, but only the tip of the iceberg.

Advanced: code style may hide branches

This is a Java-specific example, but other languages and their respective coverage tools may display a similar issue.

// These two predicates combined:
var even = n -> n % 2 == 0;     // 1 condition, 2 outcomes
var large = n -> n > 10;        // 1 condition, 2 outcomes

// Are equivalent to this single predicate:
var evenAndLarge = n -> (n % 2 == 0) && (n > 10);  // 2 conditions, 4 outcomes

However, when they are used inside list.stream().filter(predicate), the coverage tool will see either 2 or 4 branches. This will lead you to write fewer tests with weaker real coverage.

See this subpage for an in-depth explanation.

References

1. Wikipedia: Code Coverage

2. NASA: A Practical Tutorial on Modified Condition / Decision Coverage

4MB

NASA - tutorial on MCDC.pdf

PDF

Open