> For the complete documentation index, see [llms.txt](https://practical-testing.gitbook.io/home/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://practical-testing.gitbook.io/home/testing-deep-dives/computer-science-for-floating-point-arithmetic.md).

# Computer Science for Floating-point Arithmetic

For `0.1 + 0.2`, a calculator outputs `0.3` as expected. But it is quietly rounded to match your expectations. In software, decimal operations are **imprecise**. Unless special libraries (methods) are used, a program will output "odd" results. 

<figure><img src="/files/3VlVzfncgVk1GH3hOdzv" alt=""><figcaption></figcaption></figure>

These are not computer bugs. This is the inevitable consequence of:

* The fact that there are infinitely many real numbers. To represent them, we'd need an infinite amount of memory, and that's impossible.&#x20;
* Squeezing infinitely many numbers thus requires a limited set or **approximate representations**.

Logically, approximations lead to imprecise, "odd" results such as the above, which ultimately lead to bugs:

* cents (money) can get lost,&#x20;
* transactions rejected because of imprecise comparisons or&#x20;
* mismatched totals
* scientific calculations producing subtly incorrect results

This page is here to:

1. Explain relatively concisely why this happens
2. What is scientific notation (`...123e-16`) and why it shows up in computation results
3. What can be done to mitigate the problem
4. How to test for related potential bugs

### Quick Summary

Just in case, here's the shortest possible explanation:

* Even "human" formats don't always allow for exact representation
  * Think ⅓ is represented as an infinite `0.33333....` in decimal
  * Since we must cut off at some point, we lose precision.
  * Multiplying `0.33333 * 3` will yield `0.99999`, not `1`. Precision lost.
* Computers process information as `1s` and `0s`
* In binary, the same happens all the time
  * E.g., `0.1` in binary is a sequence of `1s` and `0s`, followed by an infinite sequence of `0011`
  * The `0011` must be cut off at some point - and that's where precision is lost
  * Just like in decimal, doing math with imprecise representations leads to imprecise results
  * More operations may lead to imprecision becoming bigger and bigger

That's it. The rest of the page is just a more thorough explanation of the same.

### 1. Humans vs. Computers

Humans have 10 fingers. Hence, we have 10 different digits - from `0 to 9`.

Computers, at the lowest level, process information as electric charges in a chip - either there's a charge, or none. `0 or 1`.

Summary:

* **Base 10, aka the decimal system. 0 to 9. For humans.**
* **Base 2, aka binary. 0 or 1. For computers.**

### 2. Bits and Bytes

We want to "translate" human numbers (base 10) to computer numbers (base  2). It is done like so:

<table><thead><tr><th width="113.54547119140625" align="right">Decimal</th><th width="148.99993896484375">Binary</th><th>Bits used</th></tr></thead><tbody><tr><td align="right">0</td><td>0</td><td>1 bit</td></tr><tr><td align="right">1</td><td>1</td><td>1 bit. Thus 1 bit can store 2 patterns - <code>0 or 1</code>.</td></tr><tr><td align="right">2</td><td>10</td><td>2 bits</td></tr><tr><td align="right">3</td><td>11</td><td>2 bits. <br>Thus 2 bits can store 4 patterns - <code>00, 01, 10, 11</code>.</td></tr><tr><td align="right">4</td><td>100</td><td>3 bits</td></tr><tr><td align="right">5</td><td>101</td><td>3 bits</td></tr><tr><td align="right">…</td><td>…</td><td>…</td></tr><tr><td align="right">8</td><td>1000</td><td>4 bits</td></tr><tr><td align="right">…</td><td>…</td><td>…</td></tr><tr><td align="right">16</td><td>10000</td><td>5 bits</td></tr><tr><td align="right">…</td><td>…</td><td>…</td></tr><tr><td align="right">128</td><td>10000000</td><td>8 bits. aka 1 byte.</td></tr><tr><td align="right">…</td><td>…</td><td>…</td></tr><tr><td align="right">255</td><td>11111111</td><td>8 bits. aka 1 byte. <br>Can store a number from 0 to 255 (a total of 256 numbers)</td></tr></tbody></table>

**Summary:**

* 1 bit is the smallest unit of storage
* 1 byte is the smallest **practical** unit - enough to hold at least 1 typed character ('a', '2', '$')
* Add 1 bit to double the number of patterns
* 1 bit - 2 patterns
* 2 bits - 4 patterns (e.g., `01`)
* 8 bits - 256 patterns (one byte) (e.g., `01011100`)
* Mathematically: n bits yield 2<sup>n</sup> patterns (2 to the nth power)
* **This doubling with every bit starts yielding huge numbers rather quickly**&#x20;

**Also:**

* 4 bytes (32 bits) are typically used to store integers&#x20;
  * Range −2,147,483,648 to 2,147,483,647 (billions)
* 8 bytes (64 bits) are typically used to store bigger integers (longs)
  * Range −9,223,372,036,854,775,808 to 9,223,372,036,854,775,807 (quintillions)

{% hint style="info" %}
**34 bits and 64 bits are important milestones. Keep them in mind as you'll see them later.**
{% endhint %}

**Dealing with huge numbers such as above can be cumbersome. Is there a faster, more convenient way to represent them? See the next section.**

### 3. Handling long numbers with scientific notation

It's useful, and almost necessary, to have a unified format to represent numbers. Including very small numbers and very big numbers.

* To an engineer building a highway, it does not matter whether it’s 10 meters or 10.0001 meters wide.
* On the contrary, 0.0001 meters (a tenth of a millimeter) is a *huge* difference to a microchip designer.

To satisfy the engineer and the chip designer, a number format has to provide accuracy for numbers at very different magnitudes. Also, when numbers get very big or very small, it becomes hard (and error-prone) to do math with them, even "on paper".

How much is `300000000 * 0.00000015` ?

A bit difficult, isn't it? **This can be rewritten in scientific notation.**

The format is `a * 10^n`. Here, 10 is a constant (decimal system, 10 fingers). The solution becomes much easier.

```java
// 1. Multiply 3 * 1.5 
// 2. Subtract exponents (8-7)
// 3. Result: 4.5 * 10^1 or just 45
3   * 10^8 
* 
1.5 * 10^-7
=
4.5 * 10^1    // (or just 45)
```

**The scientific notation is a fast, compact format not just for humans, but for machines too!**&#x20;

But it must be written in base 2 format - `a * 2^n`.&#x20;

Same as decimal, but we replace the constant 10 with 2.

```java
// Base 10, decimal
300,000,000 = 3 * 10^8        
0.00000015  = 1.5 × 10^-7

// Base 2, binary
300,000,000 ≈ 1.117 × 2^28        
0.00000015  ≈ 1.0111 × 2^-23
```

{% hint style="warning" %}
Notice that binary representations use ≈, i.e., the binary representation is **approximate**. This is important and will become evident later.
{% endhint %}

The general formula is expressed as:

```java
Number = coefficient * base_constant^exponent
15000  =   1.5       *   10         ^ 4
```

#### "Floating Point"

The "floating" in **floating-point** refers to the fact that the **decimal (or binary) point can "float"** — it’s not fixed in one place.

* In regular numbers, the point is fixed: `123.45` → the decimal is after the 3.
* In floating-point, you store **coefficient** **× base^exponent**, so the "point" moves depending on the exponent:

Example:

<table><thead><tr><th width="197.99996948242188">Number</th><th>Scientific / Floating-point</th></tr></thead><tbody><tr><td>123.45</td><td>1.2345 × 10² → decimal point "floats" 2 places</td></tr><tr><td>0.00123</td><td>1.23 × 10^-3 → decimal point "floats" -3 places</td></tr></tbody></table>

**Summary:**

<img src="/files/KyStDouvCKUSKOPfZxWj" alt="" class="gitbook-drawing">

All numbers can be represented in this format with any base. We generally only care about base 10 or base 2 (binary). But in binary, some fractions are only approximate. Why? Find out in the next section.

### 4. Approximate representations

There is more than one way to represent numbers. For example, fractions:

$$
\frac{1}{3} + \frac{1}{3} + \frac{1}{3} = 1
$$

However, as you may remember from school, `1/3` cannot be represented exactly in decimal, it is `0.33333333.....` infinitely repeating.&#x20;

**We cannot go on forever, so we must truncate at some point. We lose precision when we do this. How much precision we decide to lose depends on context.**

Suppose we decide to truncate after the 4th digit after the decimal.&#x20;

As such, adding `0.3333 + 0.3333 + 0.3333` will yield `0.9999`. Never a full `1`. This may be good enough for many general purposes, but not enough for manufacturing microchips.

{% hint style="info" %}
As you can see, even the decimal system fails to represent some numbers exactly.
{% endhint %}

### 5. Floating-point representation

**Binary fractions face the same problem - some numbers can’t be represented exactly, just like `1/3` in decimal.**

In binary floating-point, all integers are represented exactly only up to a fixed limit (about 9 quadrillion for double-precision). Beyond that, integers are rounded.

**However, only fractions whose denominators are powers of 2 (`1/2`, `1/4`, `3/8`, etc.) can be exactly represented in floats or doubles. All others (`0.1`, `0.22`, `0.357`, etc.) are approximations.**

For example, here is the full binary representation of `0.1`

```java
// the length is 64 bits
// notice the (infinitely) repeating pattern of 0011, until it is cut off, 
// just like we must cut off 0.333333, or we run out of memory (bits length)
0011111110111001100110011001100110011001100110011001100110011010
```

**The truncated sequence is no longer 0.1, but something very close to it.**

**These bits represent the binary scientific notation of the number. (See section 3)**

```
0         01111111011     1001100110011001100110011001100110011001100110011010
|                |                 |
Sign (+ or -)    Exponent       Mantissa / Significand
(1 bit)          (11 bits)       (52 bits)                = total 64 bits

When normalized for binary scientific notation (1.x × 2^n), the significand becomes:
1.1001100110011001100110011001100110011001100110011010 × 2^-4
```

{% hint style="info" %}
The main takeaway is that **since computers must cut off numbers at some point, precision is lost.**
{% endhint %}

### 6. Doing math with approximations

```java
// How 0.1, 0.2, and 0.3 are approximated. Slightly higher or lower.

// 0.1 with truncated bits and lost precision results in:
0.1000000000000000055511151231257827021181583404541015625

// 0.2 with truncated bits and lost precision results in:
0.200000000000000011102230246251565404236316680908203125
                  ^              
// in 64 bits, imprecision often starts around 16-17th digit after the decimal point
                   
// 0.3 with truncated bits and lost precision results in:
0.299999999999999988897769753748434595763683319091796875
```

As such, `0.1 + 0.2` actually results in:

```java
0.3000000000000000444089209850062616169452667236328125
```

We can now see that the tiny difference is big enough for the equality check to yield "false".

```java
// what's written in the program 
0.1 + 0.2 == 0.3

// the approximations the computer works with
0.3000000000000000444089209850062616169452667236328125 == 0.299999999999999988897769753748434595763683319091796875
```

You should also now understand the example at the start:

```javascript
> 2.6 - 0.7 - 1.9
2.220446049250313e-16

// Explanation:

// The output is a number very close to zero, but not quite
// also can be written as 2.22044604925031 * 10⁻¹⁶
// or as 0.0000000000000002220446049250313
```

### 7. Single vs. Double precision

More bits mean **more precision** and a **larger representable range**.

* **32-bit (single precision / `float`)**
  * Faster, uses less memory
  * \~7 decimal digits of precision
  * Rounding errors appear sooner
* **64-bit (double precision / `double`)**
  * More memory, slightly slower
  * \~16 decimal digits of precision
  * Much smaller rounding errors

Most modern programs and tests use **double precision by default**. Notice that with 64 bits, imprecision starts much further away from the decimal point.

<table><thead><tr><th width="175.3636474609375">Precision</th><th width="166.9090576171875">Example value</th><th>Stored value (approx.)</th></tr></thead><tbody><tr><td>32-bit (float)</td><td>0.1</td><td>0.10000000149011612</td></tr><tr><td>64-bit (double)</td><td>0.1</td><td>0.10000000000000000555</td></tr></tbody></table>

**Why this matters in testing:**

* The same calculation may produce different results in `float` vs `double`
* Equality checks (`==`) are **more fragile** with lower precision
* Tests that pass locally may fail on:
  * different hardware
  * GPUs vs CPUs
  * systems using single precision internally

**Expect double precision with most general-purpose programming and backend software.**

{% hint style="info" %}
A 128-bit floating-point format exists, but it is niche for very high-precision scientific computing, not typical commercial software. If you were doing rocket science, you'd probably not be on this page :).&#x20;
{% endhint %}

### 8. What to do about imprecisions

Most popular languages do not try to "fix" floating-point math. Instead, they provide tools and conventions to work *around* imprecision when it matters.

The common strategies are:

* Use **tolerances** instead of equality
* Use **specialized precision types** for money and exact math

#### Custom tolerance example

{% tabs %}
{% tab title="Python" %}

```python
a = 0.1 + 0.2
b = 0.3

epsilon = 1e-9  # Treat numbers as equal if they differ by less than 0.000000001

if abs(a - b) < epsilon:
    print("Values are equal within tolerance")
else:
    print("Values are NOT equal")

# or simply
import math
math.isclose(0.1 + 0.2, 0.3)

```

{% endtab %}

{% tab title="Java" %}

```java
double a = 0.1 + 0.2;
double b = 0.3;
double epsilon = 1e-9;
if (Math.abs(a - b) < epsilon) {
    System.out.println("Values are equal within tolerance");
} else {
    System.out.println("Values are NOT equal");
}
```

{% endtab %}
{% endtabs %}

#### Special types for decimal math

In real, robust applications, special libraries are used to avoid rounding errors. In other words, and simply put, produce results the way a calculator would.

<table><thead><tr><th width="117.18182373046875">Language</th><th>Default floating type</th><th>How imprecision is handled</th></tr></thead><tbody><tr><td>Java</td><td><code>double</code> (64-bit)</td><td>Use tolerances for comparisons; use <code>BigDecimal</code> for exact decimal math</td></tr><tr><td>C#</td><td><code>double</code> (64-bit)</td><td>Use tolerances; use <code>decimal</code> for financial / exact calculations</td></tr><tr><td>Python</td><td><code>float</code> (64-bit)</td><td>Use tolerances; use <code>decimal.Decimal</code> or <code>fractions.Fraction</code></td></tr><tr><td>JavaScript</td><td><code>Number</code> (64-bit double)</td><td>Use tolerances; use libraries (<code>BigInt</code>, decimal libs) for exact math</td></tr></tbody></table>

### 9. How to test for floating-point errors

See the [dedicated page](/home/testing-deep-dives/testing-for-floating-point-errors.md).

### References

1. Floating-point guide (simple): <https://floating-point-gui.de/formats/fp/>
2. What Every Computer Scientist Should Know About Floating-Point Arithmetic, by David Goldberg. The "classic" paper on the topic. Complex. Available on [docs.oracle.com](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html) or in the PDF below.
3. RapidTables: [Binary to Decimal converter](https://www.rapidtables.com/convert/number/binary-to-decimal.html)

{% file src="/files/2dee1lSy3Y0LogFg7j6G" %}