> For the complete documentation index, see [llms.txt](https://practical-testing.gitbook.io/home/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://practical-testing.gitbook.io/home/test-automation/code-coverage-types.md). # Code Coverage Types > **"What exactly do you mean by 80% coverage?"** ### Quick Summary * Many code coverage types exist. Some are more thorough or rigorous than others: * Least thorough: **Line, Statement, Function** * Better: **Branch / Decision** * More thorough: **Condition, Modified Condition/Decision** (MC/DC) * If you say "We have 80% code coverage", you MUST be able to elaborate or specify which type you mean if anyone asks. * **Bugs are still very likely, even with 100% code coverage of any type.** * Example: code for unexpected input or events that was simply never written (division by zero, unsupported character, abruptly terminated connection, "file not found", etc.) ### Definition and caveats {% hint style="success" %} Code coverage is a metric that shows the percentage of your code that is covered by tests. {% endhint %} {% hint style="danger" %} Code coverage does NOT tell you how good the tests are. Do they test partition boundaries? Do they include important [special values](/home/techniques/ep-and-bva.md)? Etc.\ \ Code coverage does NOT, and CANNOT tell you anything about ABSENT code - the code you've never written to handle the scenarios you've never thought of. {% endhint %} ### Line and Statement Coverage These 2 coverage types are simplest to understand, but they are also the least rigorous. They also often show identical results. ```java // Program v1: // 2 lines, 2 statements // program executes sequentially: 100% line and 100% statement coverage count++; updateTotal(count); // Program v2: // 1 line, 2 statements (separated by a semicolon) // program still executes sequentially: 100% line and 100% statement coverage count++; updateTotal(count); ``` Line and statement coverage work well with trivial lines of code with no special logic. The two metrics start diverging when branching gets involved, and 2+ statements are written on 1 line. ```java // 1 line, 2 statements // if isAdult = false, then approve() statement will never run // in this case: 100% line coverage, 50% statement coverage if(isAdult) approve(); ``` If you believe the above example is contrived, consider that Python's popular list comprehensions (an alternative to loops) are one-liners that may contain a lot of logic, including branching and value transformations. ```python # reads as "loop over items, if item X is valid, add X*X to result list" # 100% line coverage, # <100% statement coverage, depending on is_valid() implementation and value of x result = [x * x for x in items if is_valid(x)] ``` ### Function Coverage ```java // Any ONE invocation with ANY input means 100% coverage for this function int calculate(int a, int b) { // very long and complex computation // spanning 20-30 lines } ``` Similar to line and statement coverage, function coverage can only serve as an **indicator of large coverage gaps**, i.e. something that hasn't been tested at all. However, a function (or a method) with several parameters typically requires much more rigorous testing. See coverage types below. ### Control Flow Coverage Types There are many "Control Flow" coverage types, many with similar names and very fine-line distinctions: * Decision / Branch Coverage (DC) * Condition Coverage (CC) * Condition/Decision Coverage (CDC) * Modified Condition / Decision Coverage (MC/DC) * Multiple Condition Coverage (MCC) The rest of the page explains the fine differences between these. {% hint style="info" %} **TLDR**: **CDC and MC/DC are balanced, yet rigorous enough approaches for most cases.** Note that some Coverage Tools (e.g. Intellij's internal coverage counter) use them, yet still somewhat confusingly calls it "Branch". {% endhint %} Before diving into each type, consider these requirements: > Approve: if the applicant is an adult AND they have a job AND their monthly salary is over 1000 > > Reject: in all other cases Below are the graph and the code representing such a requirement. #### Requirements Graph #### Requirements Code From the graph and code, it follows:
CountWhatComment
1Code-level decisionif(decision){ }
2Business outcomes (approve / reject) and thus 2 code branches
3 (N)The decision is comprised of 3 (N) conditions aka atomic boolean expressions, predicates
4 (N + 1)Graph orange Leaf Nodes Logical end of any control flow path
6 (N*2)Graph blue branches. They also correspond to code condition outcomes.Each decision has 2 outcomes (true/false). Here 3 decisions * 2 outcomes = 6.

isAdult has 2 outcomes
hasJob has 2 outcomes
salary ≥ 1000 has 2 outcomes
8 (2^N)Number of all possible condition outcomes.Corresponds to the Full Decision Table.
### Branch / Decision Coverage **Coverage criteria**: Every **decision** in the program has **all possible outcomes** (code branches) at least once. ```java // Min. Tests needed - any combination that leads to triggering both branches // Test 1: isAdult = true; salary = 2000 // Test 2: isAdult = true; salary = 500 if (isAdult && hasJob && salary >= 1000) { approve(); // Test 1 triggers Outcome / Branch 1 } else { reject(); // Test 2 triggers Outcome / Branch 2 } ``` {% hint style="danger" %} "Branch" coverage most likely means something else in your Test Coverage tool. It definitely does in IntelliJ and other IDEs and tools by JetBrains. {% endhint %} ### Condition Coverage (CC) **Coverage criteria**: Every **condition** in a decision in the has taken **all possible outcomes** at least once ```java // For 100% coverage, code must run with: // isAdult both true and false // hasJob both true and false // salary >= 1000 both true and false if (isAdult && hasJob && salary >= 1000) { approve(); } else { reject(); } // That's potentially 3*2=6 tests, // But only 4 (N+1) tests are needed to satisfy the criteria // From the code perspective, because control flow short-circuits out of the evaluation // i.e. if isAdult=false -> other values don't matter anymore ``` From the graph perspective, consider that one test can cover many edges (graph branches). ### Condition / Decision Coverage (CDC) **Coverage criteria**: as the name implies, it is simply the **combination of Condition + Decision coverage**. In this simple example, and in many cases, Condition coverage naturally leads to Decision coverage as well. ### Modified Condition / Decision Coverage (MC/DC) **Coverage criteria**: **Every condition** in a decision **has shown to independently affect** decision's **outcome.** In this example, while holding other conditions constant, we must demonstrate that: * `isAdult` independently affects the decision * `hasJob` independently affects the decision * `salary >= 1000` independently affects the decision In this example, again, CC, CDC and MC/DC can all be satisfied with the same set of 4 tests. MC/DC distinction becomes more apparent when conditions are **masked**, **redundant**, or **logically subsumed (e.g. A && (B ||C) ).** For a more thorough explanation, see this [subpage tutorial](/home/test-automation/code-coverage-types/dc-vs.-cc-vs.-cdc-vs.-mc-dc.md). For a more general demonstration, see [Elementary Comparison](/home/techniques/elementary-comparison.md). ### Multiple Condition Coverage (MCC) Coverage criteria: **Every combination** of condition outcomes within a decision has been invoked at least once. With 3 conditions, we get 2^3=8 combinations, thus 8 tests. This corresponds going through a full, non-collapsed [Decision Table](/home/techniques/decision-tables-and-trees.md). This is the most thorough, yet rarely practical approach, because the number of combinations grows exponentially. (6 conditions already leads to 2^6=64 tests). ### "Branch" Coverage in Tools Many tools may report "Branch" Coverage, but it actually means something else. {% hint style="danger" %} Take a moment and understand what your tool means by "Branch", so you understand the strengths and weaknesses of the underlying approach. {% endhint %} For example, IntelliJ IDEA (Java) doesn't look at source code conditions (1), NOR at source code conditional "branches", but at [**bytecode branches**](https://en.wikipedia.org/wiki/List_of_Java_bytecode_instructions) (2). There, it sees 3 branches, each with 2 possible outcomes, hence it counts 6 branches (3) to cover.
**In practice, however, bytecode branch coverage + statement coverage should almost always correspond to at least Condition / Decision Coverage (CDC) or even MC/DC.** As such, the coverage tools for interpreted languages such as JavaScript or Python should offer a rigorous underlying coverage type. {% hint style="info" %} It's best to know what exactly your tool means by "branch coverage" to know its strengths and weaknesses. {% endhint %} ### Where 100% Coverage Fails #### Off-by-one errors Control flow coverage is not concerned with test data quality. For `salary > 1000`, it is enough to select random values, such as `500` and `2000`, to achieve "full branch / condition coverage". But if the requirements said "salary 1000 or more", then `salary > 1000`, clearly has a bug, it should be `salary >= 1000`. This example demonstrates that **coverage metrics are blind to the gap between source code and requirements**, implicit or explicit. Border values are better - at least `999` and `1000`, though `1001` could be added for additional thoroughness. {% hint style="success" %} Improve tests driven by "branch coverage" with better test data using [EP & BVA](/home/techniques/ep-and-bva.md) {% endhint %} #### The code that never was Internally, some function `process(input1,input2,input3)` might fetch a file over the network, get a value from the file, use a complex formula to process the inputs, and save the result to another file. With 100% function, statement, and branch coverage (even MCC - the most exhaustive one), the following bugs occur: 1. :bug: Oops, the network connection failed. 2. :bug: Oops, the file wasn't there 3. :bug: Oops, the file was empty or had corrupted values 4. :bug: Oops, the value calculated by one function was 0, given to another function, and a division by zero error occurred 5. :bug: Oops, the sum (result of a multiplication) turned out to be bigger than what an `Integer` variable can hold, so either rounding happened, or an exception was raised 6. :bug: Oops, the result failed to be saved to a file (full disk, no permission, other) 7. :bug: Oops, the result was saved too late, causing a dependent service to fail when it tried to consume it. 8. :bug: Oops, one of the requested features was overlooked and not implemented! All of the above and many more failures might occur because no code was written for such scenarios. No one ever thought of them! **Coverage metrics, by definition, can only evaluate what is in code, and many precautions might be missing. It could be argued that most challenging bugs occur at the integration level.** {% hint style="info" %} High code coverage with carefully picked test values is a good start, but only the tip of the iceberg. {% endhint %} ### Advanced: code style may hide branches This is a Java-specific example, but other languages and their respective coverage tools may display a similar issue. ```java // These two predicates combined: var even = n -> n % 2 == 0; // 1 condition, 2 outcomes var large = n -> n > 10; // 1 condition, 2 outcomes // Are equivalent to this single predicate: var evenAndLarge = n -> (n % 2 == 0) && (n > 10); // 2 conditions, 4 outcomes ``` However, when they are used inside `list.stream().filter(predicate)`, the coverage tool will see either 2 or 4 branches. This will lead you to write fewer tests with weaker real coverage. See this [subpage for an in-depth explanation](/home/test-automation/code-coverage-types/functionally-identical-code-different-branch-count-java.md). ### References 1. [Wikipedia: Code Coverage](https://en.wikipedia.org/wiki/Code_coverage) 2. [NASA: A Practical Tutorial on Modified Condition / Decision Coverage](https://ntrs.nasa.gov/api/citations/20010057789/downloads/20010057789.pdf) {% file src="/files/O0AM1WnxseamaGmKKWJr" %}