> For the complete documentation index, see [llms.txt](https://practical-testing.gitbook.io/home/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://practical-testing.gitbook.io/home/techniques/pairwise-testing.md). # Pairwise Testing ### Quick Summary * Coupled with domain partitioning, Pairwise Testing is an effective technique to reduce the combinatorial explosion to a manageable set of tests * The greater the exhaustive test set, the greater the savings with Pairwise (e.g., reduce from 40 tests to 20, or 2400 to 80). * Use PICT or other tools to generate pairs * Pairwise can be used for both Input Values and Configuration or Environment parameters. * Pairwise output is not perfect - review it, remove non-meaningful or low-value combinations, and add high-value ones. Use domain knowledge and expertise to do so. ### Testing in Isolation vs. in Combination [EP & BVA](/home/techniques/ep-and-bva.md) techniques typically apply to a single input. In other words, inputs (or configurations, or others) are tested **in isolation**. This must be done, but it is far from enough. Even relatively simple software has multiple **variables that interact**. These variables may come from the UI, the operating system, configuration files, a database, the network, or other sources. Many bugs arise from such interactions. In combinatorial testing, the task is to verify that the software functions correctly with such variable combinations. {% hint style="info" %} Many bugs arise from interactions of multiple variables, regardless of their origin. {% endhint %} ### Calculating Combinatorial Explosion The first task is to handle (reduce) the number of test cases, which **grows exponentially** with the number of variables and possible values. Q: How to tell how many combinations I am facing? The **basic formula** for counting total combinations is: $$ Total combinations=v1​×v2​×v3​×⋯×vn​ $$ Where: * v1, v2, vn = the number of possible values for each variable. * You multiply them together **Example 1:** Given 3 checkboxes (binary `on/off` values only), then it is the same as the growth of a [Decision Table](/home/techniques/decision-tables-and-trees.md) (23 or more generally 2n) $$ Total combinations=2×2×2=8 $$ **Example 2:** Some variables have 3+ values * **Browser**: 3 values (Chrome, Firefox, Safari) * **OS**: 2 values (Windows, macOS) * **Language**: 4 values (EN, FR, ES, DE) $$ Total combinations=3×2×4=24 $$ **Example 3:** Booking a hotel, **multi-select possible**, one input field accepts a range of integers * **Hotel**: 3, 4, or 5 stars * **Who is staying**: Adults only, adults with kids * **Add-ons (multi-select, 4 options, so 2****4**** combinations)**: unlimited buffet (yum!), late checkout, airport transfer, parking * **Nights (integer)**: 1-10 values allowed $$ Total combinations=3×2×2^4×10=960 \text{ (without considering invalid values) } $$ {% hint style="info" %} For multi-select dropdowns with "nothing selected" allowed, the formula is 2k (k is the number of options) {% endhint %} For mission non-critical systems, 960 tests are too many. How can we reduce the number intelligently but retain good coverage? ### Step 1: Start with Domain Partitioning Apply [EP & BVA](/home/techniques/ep-and-bva.md) techniques. The primary candidate in the hotel booking is **Nights.** We can reduce the 1-10 values to: * 1 and 10 (min, max) * 1, 5, and 10 (min, mid, max) {% hint style="info" %} We should apply our experience and domain knowledge to identify suitable variables. Identifying **default, frequent, or commonly used values** improves the quality of our tests. {% endhint %} Suppose we find out that: * 7 nights (a week) is the most common length of stay * Almost no one ever chooses "Late Checkout" Thus, we may decide to: * Choose 1, 7, and 10 (min, common, max) values for "Nights" * Leave out "Late Checkout" from the combinations. We can test it separately later $$ Total combinations=3×2×2^3×3=144 $$ We are down to 144 cases thanks to EP, BVA, and domain knowledge. We may now proceed to combinatorial testing. ### Step 2: Understand All Singles (Optional) The "All Singles" is rarely used in practice, as it reduces the number of cases (and coverage) too much. It serves as a stepping stone to the Pairwise technique. "Full coverage" in All Singles means that **every value of every variable appears in at least one test**, but we don’t care about combinations with other variables. To achieve **all singles**, you only need **at least as many tests as the largest number of values in any variable.** $$ All ; Singles ;Cases = largest ;number ;of; values ; in ;any; variable $$ Normally, we just pick the highest number. In our previous formula of **3×2×2****3****×3,** it would normally be **2****3****= 8, resulting in 8 cases**. But since this is a multi-select dropdown, 8 represents combinations, not distinct values. There are only 3 distinct values. Thus, 3 is our highest number. | Test | Hotel Stars | Who is Staying | Add-ons | Nights | | ---- | ----------- | -------------- | ---------------- | ------ | | 1 | 3 | Adults only | Buffet | 1 | | 2 | 4 | Adult + Child | Airport Transfer | 7 | | 3 | 5 | Adults only | Parking | 10 | This table uses every possible value of every variable, but as you might guess, there is a high chance of missing a combinatorial bug with just 3 out of 144 cases. ### Step 3: Achieve All Pairs (Pairwise) All Pairs, aka Pairwise technique, is much more robust than All Singles, yet keeps the number of tests manageable. While there is no universal formula, a rule of thumb is to simply multiply the two largest numbers of values in any variable. $$ All ; Pairs ; Cases = \newline largest ;number ;of; values ; in ;any; variable × \newline second ; largest ;number ;of; values ; in ;any; variable $$ Examples:
What's Being TestedAll CombinationsAll Pairs Case #All Singles Case #

3 browsers x

2 Operating systems x

4 languages

3 x 2 x 4 = 244 x 3 = 12
(saves 50%)		4
5 car brands x
4 colors x
2 engine types		5 x 4 x 2 = 405 x 4 = 20
(saves 50%)		5
3 subscription plans x
5 payment methods x
5 regions		5 x 5 x 3 = 755 x 5 = 25
(saves 66%)		5
Var 1 with 10 values x
Var 2 with 8 values x
Var 3 with 6 values x
Var 4 with 5 values		10 x 8 x 6 x 5 = 240010 x 8 = 80
(saves 96%)		10
{% hint style="success" %} The greater the number of All Combinations, the greater are the savings from Pairwise {% endhint %} ### Step 4: Create Pairwise Table #### Available Tools It is possible to create the table manually, but it's a tedious and error-prone process. Instead, use a tool. Both commercial and open-source alternatives exist. PICT is a widely used open-source tool: * GitHub by Microsoft: * Online - an online service powered by Microsoft PICT created by an individual: #### Quick PICT Tutorial for Windows Users Non-Windows users: see [instructions](https://github.com/microsoft/pict?tab=readme-ov-file#building-on-linux-osx-bsd-etc) on how to build the tool from source. For Windows users: 1. Go to the [releases](https://github.com/microsoft/pict/releases/) page and download the latest .exe binary 2. PICT is a CLI tool, so double-clicking the .exe achieves nothing. Instead, open the `cmd` and run it ``` >pict.exe Pairwise Independent Combinatorial Testing Usage: pict model [options] Options: /o:N|max - Order of combinations (default: 2) /d:C - Separator for values (default: ,) /a:C - Separator for aliases (default: |) /n:C - Negative value prefix (default: ~) /e:file - File with seeding rows /r[:N] - Randomize generation, N - seed /c - Case-sensitive model evaluation /s - Show model statistics ``` PICT is a sophisticated tool that allows a lot of behavior fine-tuning, and the [documentation](https://github.com/Microsoft/pict/blob/main/doc/pict.md) is very well written. Below is just a basic example of how to use it. The only mandatory argument that the CLI expects is the `model` - a plain text file in a specific format. Assuming the model file is `input.txt` , is located in the same dir as the .exe file, and it contains: ``` Browser: Chrome, Firefox, Edge OS: Windows, Mac UserType: Admin, Guest ``` When you run: ``` pict.exe input.txt > output.txt ``` Then the output.txt should contain: ``` Browser OS UserType Chrome Mac Guest Chrome Windows Admin Edge Mac Admin Firefox Windows Guest Firefox Mac Admin Edge Windows Guest ``` You can now use the file to test software manually, or feed it to an automation script. ### Step 5: Review and Amend the Table #### Changing non-meaningful pairs Using the Pairwise technique is like throwing everything into a mixer. As such, the output is not guaranteed to be always meaningful. Non-meaningful examples: ``` Product Delivery Method Payment Method -------------------------------------------- E-book Download Cash on Delivery (can't pay cash for a download) Flight Passenger Bags ------------------------------------------ Return 1 Child 3 (child can't fly alone) OS CPU DotNetVersion ------------------------------------------ Ubuntu AMD 4.8 (no .NET on Ubuntu)* ``` > \*Example is from [PICT tool docs](https://github.com/Microsoft/pict/blob/main/doc/pict.md) Review the output table and either manually adjust it or use the tool to prevent undesirable combinations. PICT allows you to do this with special syntax: ``` OS: Windows, Ubuntu CPU: Intel, AMD DotNetVersion: 4.8, 4.8.1, NA IF [OS] = "Ubuntu" THEN [DotNetVersion] = "NA" ELSE [DotNetVersion] <> "NA"; ``` #### Adding special or valuable pairs [**Step 1**](#step-1-start-with-domain-partitioning) emphasizes the importance of utilizing domain knowledge to generate more effective test values for domain partitioning. The same principle applies when reviewing the Pairwise Table. Use your domain knowledge to: * **remove** meaningful but low-priority combinations, * **add** high-value, frequent, commonly used, critically important combinations Again, the tool you're using should allow you to fine-tune the output without having to amend it later. ### Use Pairwise for Both Input Values and Configurations The content on this page had several examples, one with Hotel selection, and another for combinations of Browsers x Operating Systems x etc. It is no coincidence. Pairwise testing can be applied to: 1. Business or technical **input values** (coming from UI, API, or other) 2. Environmental or configuration **parameters**: hardware, OS, browser, tech stack version, protocol, encoding, encryption, etc. ### FAQ TLDR: any question you may have in relation to the practicalities of using a combinatorial tool should be covered in the documentation of your tool. Nevertheless, here are the more important ones: Q: How do I prevent the tool from creating invalid or ineffective combinations? A: The tool should support constraint creation. PICT allows both simple and non-trivial constraints. Q: I want the test suite to always contain certain important combinations. A: This is called a "seed". PICT allows this. See [Seeding](https://github.com/Microsoft/pict/blob/main/doc/pict.md) section. Q: I want to have "negative" scenarios with invalid values, but I don't want to have any scenarios with multiple invalid values because of "input masking", i.e. when the 1st invalid value is handled, the effect of the 2nd invalid value (or all other values) is typically "masked". How do I go about this? A: Two solutions: 1. Use a separate input model and create a separate set with negative scenarios only 2. If the tool allows, use special syntax to mark invalid values. The tool can then ensure that only one invalid value per row is used. See PICT's [Negative Testing](https://github.com/Microsoft/pict/blob/main/doc/pict.md#negative-testing) section. Q: I have determined that some parameter groups must be covered at higher interaction strength `t` (3-way, 4-way, instead of 2-way (pairwise)). But if I increase the `t` on the entire input model (ie ALL parameters), I incur the penalty of having many more test cases. Is there a balanced approach? Example (plain): ``` Parameters A, B, C → 3-way coverage required All other parameter pairs → 2-way coverage ``` A: Yes, it is called *mixed-strength arrays*. If the tool allows it, you may specify higher interaction strength just for the selected parameters. See PICTs [Sub-Models](https://github.com/Microsoft/pict/blob/main/doc/pict.md#sub-models) section. ### Criticisms In their paper "Pairwise Testing: A Best Practice That Isn’t", Bach and Shroeder provide important criticisms and caveats on how and why Pairwise Testing may fail. The proposed solutions below are not part of the original paper. 1. **Pairwise testing fails when you don’t select the right values to test with.** 1. If prior Domain (Equivalence) Partitioning was done poorly, then the weakness of such selected values is carried over into pairwise testing. 2. Solution: carry out high-quality [EP\&BVA](/home/techniques/ep-and-bva.md), construct your own scenarios of highest importance, and provide them as seed to the Pairwise tool 2. **Pairwise testing fails when you don’t have a good enough** [**oracle**](https://practical-testing.gitbook.io/home/bug-advocacy/craft-a-useful-bug-report#oracles-and-hiccups)**.** 1. The value pairs (or triples, or quadruples) chosen DO expose a fault, but you or the automated test script fails to notice it. The fault may be unforeseen and hidden - gradual data or state corruption, an error in a downstream service that is not being observed as part of the scoped test. 2. Solution: There is no single solution or "best practice" for this other than using one's best judgment. 3. **Pairwise testing fails when highly probable combinations get too little attention.** 1. This is similar to Point 1 - in addition to EP\&BVA, domain knowledge may be used to select highly used or highly probable value combinations (usage statistics, "factory" default configurations, etc.) 4. **Pairwise testing fails when you don’t know how the variables interact.** 1. There is plenty of software that produces output based on 3,4 and even 10 inputs. In such cases, Pairwise (2-way) testing is not very likely to find bugs hiding in such high-strength interactions. 2. Solution: analyze the software (from both black-box and, if possible, white-box perspectives) to determine the most appropriate strength of combinatorial testing and produce mixed-strength arrays. For example: 1. Area A of SUT uses variables A-D, max. Interaction is 2 - apply Pairwise testing 2. Area B of SUT uses variables E-M, max. Interaction is 5 - consider applying 4- or 5-way testing. ### References 1. PICT [documentation](https://github.com/Microsoft/pict/blob/main/doc/pict.md) 2. Pairwise Testing in the Real World {% file src="/files/84X9X7Kt1vPId25W83Vd" %} 3. Pairwise Testing: A Best Practice That Isn’t {% file src="/files/FQL9QsEbvghD8ngX1uIB" %}