---
title: Decision measures explained
summary: What the probability, the lift range, and the other decision measures mean, and how to check each one by hand.
order: 10
---

## Overview

A 95% range says how big a difference might be and how sure the test is about it. Choosing among options also asks what each choice gives up, and what knowing more would be worth. This page computes each decision measure from one example and says what each measure does not tell you.

## What every measure is built from {#draws}

Every measure on this page is an average over draws. One draw is a single possible state of the world: a click rate for each option that fits what the test has seen so far. A test that has run longer produces more draws, and each one is still a plausible version of the truth.

A measure averages over every draw, including the draws where its option does not win. Averaging only over the draws where an option wins gives a larger number, which overstates the option, so no figure on this page is taken that way.

Real results use thousands of draws. This page uses three, because three options and three draws make nine values, and nine values let you check every figure here by hand.

## The example {#example}

Example data, not a real brand. A brand sends a campaign to 300,000 recipients a month. Its subject line today gets a click rate near 3.1%, and it tests two more: an urgency line and a curiosity line. The three options are today's subject line, urgency, and curiosity.

The table gives each option's click rate in each draw. A click rate of 3.5% is 35 clicks per 1,000 recipients.

| Draw    | Today's | Urgency | Curiosity | Best in this draw |
| ------- | ------: | ------: | --------: | ----------------- |
| 1       |    3.0% |    3.5% |      3.1% | Urgency           |
| 2       |    3.0% |    3.7% |      3.2% | Urgency           |
| 3       |    3.3% |    3.0% |      3.6% | Curiosity         |
| Average |    3.1% |    3.4% |      3.3% |                   |

Urgency has the highest average, 3.4%, so it is the leader. The best click rate in each draw is 3.5%, 3.7%, and 3.6%, which averages 3.6%. Every measure below is built from those facts, and [the example ledger](#the-example-ledger) collects them in one table.

At 300,000 recipients a month, 1 click per 1,000 recipients is 300 clicks a month. Every monthly figure on this page is the per-1,000 figure times 300.

### The example ledger {#the-example-ledger}

This table is the source of truth for every figure on the page. All values are clicks per 1,000 recipients except the chances. "A month" is the average times 300.

| Measure                                     | Option    | Draw 1 | Draw 2 | Draw 3 | Average |       A month |
| ------------------------------------------- | --------- | -----: | -----: | -----: | ------: | ------------: |
| Click rate                                  | Today's   |     30 |     30 |     33 |      31 |               |
| Click rate                                  | Urgency   |     35 |     37 |     30 |      34 |               |
| Click rate                                  | Curiosity |     31 |     32 |     36 |      33 |               |
| Best in the draw                            |           |     35 |     37 |     36 |      36 |               |
| Expected regret                             | Today's   |      5 |      7 |      3 |       5 |  1,500 clicks |
| Expected regret                             | Urgency   |      0 |      0 |      6 |       2 |    600 clicks |
| Expected regret                             | Curiosity |      4 |      5 |      0 |       3 |    900 clicks |
| Expected gain                               | Today's   |      0 |      0 |      0 |       0 |      0 clicks |
| Expected gain                               | Urgency   |      4 |      5 |      0 |       3 |    900 clicks |
| Expected gain                               | Curiosity |      0 |      0 |      3 |       1 |    300 clicks |
| Margin over the best of the others          | Today's   |     −5 |     −7 |     −3 |      −5 | −1,500 clicks |
| Margin over the best of the others          | Urgency   |     +4 |     +5 |     −6 |      +1 |   +300 clicks |
| Margin over the best of the others          | Curiosity |     −4 |     −5 |     +3 |      −2 |   −600 clicks |
| Difference, urgency minus curiosity         |           |     +4 |     +5 |     −6 |      +1 |   +300 clicks |
| Chance of direction, urgency over curiosity |           |      1 |      1 |      0 |     67% |               |
| What knowing which is best is worth         |           |      0 |      0 |      6 |       2 |    600 clicks |
| Expected improvement over urgency           | Today's   |      0 |      0 |      3 |       1 |    300 clicks |
| Expected improvement over urgency           | Urgency   |      0 |      0 |      0 |       0 |      0 clicks |
| Expected improvement over urgency           | Curiosity |      0 |      0 |      6 |       2 |    600 clicks |
| Chance of being best                        | Today's   |      0 |      0 |      0 |      0% |               |
| Chance of being best                        | Urgency   |      1 |      1 |      0 |     67% |               |
| Chance of being best                        | Curiosity |      0 |      0 |      1 |     33% |               |

The margin rows are not a measure the page names. They exist to check expected gain against expected regret, and the [checks below](#checks-the-example-passes) use only their averages.

### Checks the example passes {#checks-the-example-passes}

Each check was made by hand against the definitions behind each measure.

1. **Expected regret is the average best minus the option's own average.** 36 − 31 = 5, 36 − 34 = 2, 36 − 33 = 3. These match the averages in the ledger.
2. **The lowest expected regret belongs to the highest average.** Urgency has the highest average, 3.4%, and the lowest expected regret, 2.
3. **Each option's expected regret is the leader's plus its gap behind the leader.** Today's: 2 + (34 − 31) = 5. Curiosity: 2 + (34 − 33) = 3. Urgency: 2 + 0 = 2.
4. **What knowing is worth equals the leader's expected regret.** The average best minus urgency's average is 36 − 34 = 2, the same as urgency's expected regret row.
5. **The leader's expected improvement is 0**, because in every draw urgency is compared with itself.
6. **No expected improvement exceeds the leader's expected regret.** Today's is 1 and curiosity's is 2, and neither is above 2. Curiosity's equals it because curiosity is the best option in the one draw where urgency is not. Today's is lower because in that draw it beats urgency but not curiosity.
7. **Expected gain minus expected regret is the average margin over the best of the others.** Urgency: 3 − 2 = +1. Curiosity: 1 − 3 = −2. Today's: 0 − 5 = −5. Each matches its margin row.
8. **The average difference is the difference of the averages.** Urgency minus curiosity averages +1, and 34 − 33 = 1.
9. **The chances of being best sum to 100%.** 0% + 67% + 33%. Two thirds and one third, rounded for print.

The next three measures are report measures: they say what each option gives up, what a choice gains, and which way a difference goes.

## Expected regret {#expected-regret}

**What it is.** What choosing this option gives up on average against whichever option is truly best, given the tests so far.

**How it is computed.** In each draw, subtract the option's click rate from the best click rate in that draw, then average over all draws.

| Option    | Draw 1 | Draw 2 | Draw 3 | Average |      A month |
| --------- | -----: | -----: | -----: | ------: | -----------: |
| Today's   |      5 |      7 |      3 |       5 | 1,500 clicks |
| Urgency   |      0 |      0 |      6 |       2 |   600 clicks |
| Curiosity |      4 |      5 |      0 |       3 |   900 clicks |

Today's subject line gives up 5 clicks per 1,000 recipients, about 1,500 clicks a month. Urgency gives up 2, about 600 a month. Curiosity gives up 3, about 900 a month.

Every option's expected regret is urgency's 2 plus how far its average sits behind urgency's. Today's is 2 + 3 = 5, and curiosity's is 2 + 1 = 3. So the order of expected regret is the order of the averages, and the leader has some too.

**What it does not say.**

- That the leader is safe. Urgency's expected regret is 2, not 0, because urgency is not best in draw 3. The leader carries the uncertainty about which option is best, and every other option carries that plus its distance behind.
- That clicks were lost. Nothing has been sent yet. It is what a choice gives up on average, measured against what we know now.
- That staying put is free. Today's subject line has the largest expected regret in the example, about 1,500 clicks a month.
- That it is fixed. More tests change it, because they change what we know, not because the options change.

## Expected gain {#expected-gain}

**What it is.** How much this option wins by, averaged over every draw, including the draws where it does not win.

**How it is computed.** In each draw where the option is best, take how far it is ahead of the next best. In every other draw, take 0. Average over all draws.

| Option    | Draw 1 | Draw 2 | Draw 3 | Average |    A month |
| --------- | -----: | -----: | -----: | ------: | ---------: |
| Today's   |      0 |      0 |      0 |       0 |   0 clicks |
| Urgency   |      4 |      5 |      0 |       3 | 900 clicks |
| Curiosity |      0 |      0 |      3 |       1 | 300 clicks |

Urgency gains 3 clicks per 1,000 recipients, about 900 clicks a month. Curiosity gains 1, about 300 a month. Today's subject line gains 0.

Expected gain minus expected regret is the option's average margin over the best of the others. Urgency, averaged over every draw: about +900 clicks a month from the draws where it is best, about −600 clicks a month from the draws where it is not. The two net to about +300 a month. For curiosity they net to about −600, and for today's subject line to about −1,500.

**What it does not say.**

- Which option to choose. It rewards width: an option with a wide range wins big in a few draws, so it can lead on expected gain while giving up clicks on average. That is why expected regret is the better guide to what a choice costs.
- That the choice is settled. Expected gain does not shrink to zero as data grows; it settles at the true best option's real margin.
- How much the option wins by when it wins. The figure averages in the draws where it does not win. Averaging only the draws where it wins gives a larger number, which overstates the option, so this page prints no such figure.

## Chance of direction {#chance-of-direction}

**What it is.** How often a difference between two options is above zero, across all the draws.

**How it is computed.** In each draw, subtract curiosity's click rate from urgency's, and count 1 when the result is above zero. Average over all draws.

| Value                                       | Draw 1 | Draw 2 | Draw 3 | Average |     A month |
| ------------------------------------------- | -----: | -----: | -----: | ------: | ----------: |
| Difference, urgency minus curiosity         |     +4 |     +5 |     −6 |      +1 | +300 clicks |
| Chance of direction, urgency over curiosity |      1 |      1 |      0 |     67% |             |

The differences are +4, +5, and −6 clicks per 1,000 recipients, so the counts are 1, 1, 0. That is a 67% chance urgency beats curiosity on click rate. The average difference is +1 per 1,000, about +300 clicks a month.

**What it does not say.**

- How big the difference is. Two draws favor urgency by a little and one favors curiosity by more, so a 67% chance sits on an average of only +1 per 1,000.
- Which option is best. It compares two named options and ignores the rest. A third option can beat both.
- That the other 33% is a coin toss. Draw 3 favors curiosity by 6 per 1,000, more than either draw that favors urgency.

The next three measures are recommendation measures: they answer which features to test, which options to focus on, and how to split.

## What knowing which option is best is worth {#value-of-knowing}

**What it is.** What learning for certain which option is best would be worth, given the tests so far. It is the leader's expected regret, read as a question about the future.

**How it is computed.** In each draw, subtract the leader's click rate from the best click rate in that draw, then average. Urgency is the leader, so the values are 0, 0, and 6, and the result is 2 clicks per 1,000 recipients.

| Measure                             | Draw 1 | Draw 2 | Draw 3 | Average |    A month |
| ----------------------------------- | -----: | -----: | -----: | ------: | ---------: |
| What knowing which is best is worth |      0 |      0 |      6 |       2 | 600 clicks |

Knowing for certain which subject line is best would be worth about 600 clicks a month. This is the same 2 as urgency's expected regret.

**What it does not say.**

- That urgency is probably not best. The figure is 0 in both draws where urgency is best. It is the value of knowing, not a likely shortfall, and this page never words it as the leader trailing the true best.
- That the test is unfinished. It is a figure about what more knowledge is worth, not a verdict on the result.
- What one more test would earn. It is what perfect knowledge would be worth, which is the most any further testing could be worth. A real test learns less than everything, and it pays for what it learns with the sends that go to the weaker options.
- Anything across metrics. It compares features judged on the same metric: a feature worth 600 clicks a month is a more open choice than one worth 60.

## Expected improvement {#expected-improvement}

**What it is.** How much this option might beat today's leader by, averaged over every draw.

**How it is computed.** In each draw, subtract the leader's click rate from the option's, keep the result when it is above zero, and take 0 otherwise. Average over all draws.

| Option    | Draw 1 | Draw 2 | Draw 3 | Average |    A month |
| --------- | -----: | -----: | -----: | ------: | ---------: |
| Today's   |      0 |      0 |      3 |       1 | 300 clicks |
| Urgency   |      0 |      0 |      0 |       0 |   0 clicks |
| Curiosity |      0 |      0 |      6 |       2 | 600 clicks |

Curiosity might add 2 clicks per 1,000 recipients, about 600 clicks a month. Today's subject line might add 1, about 300 a month. Urgency adds 0, because it is compared with itself. The options worth focusing on could add up to about 2 clicks per 1,000 recipients, about 600 clicks a month.

No option's expected improvement exceeds what knowing is worth, 2 per 1,000. Curiosity's equals it, because curiosity is the best option in the one draw where urgency is not. Today's is lower: in draw 3 it beats urgency by 3 but trails curiosity.

**What it does not say.**

- A likely lift. Curiosity's figure is an average of 0, 0, and 6. In two of the three draws it adds nothing.
- A total across options. The figures do not add: 1 + 2 = 3 is more than the 2 that knowing is worth, because two options can beat the leader in the same draw.
- The chance of beating the leader. A wide option can score high on a small chance, and a narrow one low on a large chance.

## Chance of being best {#chance-of-being-best}

**What it is.** How often this option is the best one, across all the draws.

**How it is computed.** In each draw, count 1 for the best option and 0 for the rest, then average.

| Option    | Draw 1 | Draw 2 | Draw 3 | Average |
| --------- | -----: | -----: | -----: | ------: |
| Today's   |      0 |      0 |      0 |      0% |
| Urgency   |      1 |      1 |      0 |     67% |
| Curiosity |      0 |      0 |      1 |     33% |

Urgency is best in two draws of three, about 67%. Curiosity is best in one, about 33%. Today's subject line is best in none, 0%. With three options, even odds would be about 33% each.

Sending each option a share in proportion to its chance of being best is the rule known as Thompson sampling. The earn-and-learn setting raises each chance to a power and rescales the results to 100%: a setting that leans toward earning concentrates the split, and one that leans toward learning spreads it. At the middle setting the power is 1, so the split is the chance itself: 67% to urgency, because it is best in two of three draws; 33% to curiosity; none to today's subject line. In the ad version of a test, the same split decides how much budget each version gets.

On Meta, Deciding Data makes these budget changes in your ad account. In Klaviyo it only reads: it never changes a send, a split, or a flow. Your team can make the same move there, such as giving each subject line the share of the list above, and Deciding Data reads the results back out.

**What it does not say.**

- How much is at stake. Urgency is best in two draws of three, yet its average margin over the best of the others is only +1 per 1,000. A high chance can attach to a tiny difference, which is why expected regret is the better guide to how much is at stake.
- When the choice is settled. On a true tie, the chance hovers near even odds however much data comes in.
- That today's subject line can never be best. The 0% comes from having only three draws. With thousands of draws, a chance that small is rarely exactly zero.
- Anything on its own about a large test. Among ten options, even odds are 10%, so a strong option can sit at 35%.

**Where you see it.** This measure sets the suggested split in a test recommendation.

## How the measures change as tests add up {#as-tests-add-up}

As a test runs, the draws narrow, and each measure moves in the direction of its own question.

| Measure                             | As tests add up                                                                          |
| ----------------------------------- | ---------------------------------------------------------------------------------------- |
| Expected regret                     | Goes to zero for the true best option, and settles at the true shortfall for every other |
| Expected gain                       | Settles at the true best option's real margin                                            |
| Chance of direction                 | Goes to 100% or 0% when the true difference is not zero                                  |
| What knowing which is best is worth | Goes to zero, so a settled feature drops down the list of features to test               |
| Expected improvement                | Goes to zero for every option, because nothing is left to find                           |
| Chance of being best                | Goes to 100% for a clear best; stays near even odds on a tie                             |

Every measure here is computed from the same draws as the 95% range, so no measure can disagree with the range beside it.
