Deciding Data

Why Run A/B Tests?

March 29th 2022
Two beakers with green liquid

Ad platforms adjust many features of a campaign (placement, format, detailed targeting, etc.). So when running different ads, it is very hard to tell the cause of a performance difference.

A/B tests (randomized controlled experiments) help you handle these many changes and discover what matters to your ad's performance.

Marketing content matters

In 2017 Nielson released a big study of what matters in marketing. They found:

"...creative remains the undisputed champ in terms of sales drivers..."

Some content produces interest, and clicks, many times faster than others. With competition in online advertising only getting stronger, these content differences can determine whether an ad is a profitable engine of growth or a money pit.

Finding the right content is hard

Though I've had the privilege to work with some truly gifted marketers and designers, I have yet to meet anyone who can predict what will work before trying it.

But, even trying creative is challenging. When we run two ads, the ad platforms adapt many things and create many differences. Perhaps ads receive impressions at different times. Perhaps the ads get different placements on the platforms, which has a notable impact on clicks. Maybe the audience that sees the ads are different. All of these changes affect performance.

Causal Diagram of Ad Performance

When many things change at once, it takes a lot more work to understand the impact of each component.

Experiments to the rescue

To understand what we should do to get the best performance, we need a way to limit the influence of some factors. Here is where the randomization in an A/B test really helps. Randomization allows us to break some causal connections, while also allowing us to measure our certainty when estimating others.

Causal Diagram of a Randomized Ad Experiment


Using experiments well is like finally getting the right glasses. The world comes into focus. This allows us to adapt our ads and better understand what our customers want.

Deciding Data helps make ad testing easy and understandable. If you would like a tool to help understand your ads. We're happy to chat.