How Our Eyes Taught Me About Smarter Ad Tests

Hi, I'm a statistician here at Deciding Data. I help figure out why some ads work and others don't. Let's talk about how we find clarity in the chaos.

The Challenge: Identifying True Ad Performance in a Dynamic Market

When testing ads for our clients, we work to determine which ad changes truly drive results in a constantly shifting marketplace.

The world doesn't pause while we test. Competitors launch campaigns, news breaks, and trends evolve—all potentially affecting how ads perform. To make this even harder, many of the things we want to understand are delayed (customers often take time deciding if they want to purchase).

Randomized experiments help us isolate the impact of specific changes, but they come into focus gradually, while the world moves. This leaves us with a fundamental challenge of discovery:

How do we:

• Test variants enough to learn what truly works?
• Prevent our results from blurring in a constantly changing world?
• Use discoveries quickly to maximize performance?

This is our continuous balancing act: gaining reliable insights in a dynamic environment while simultaneously maximizing performance—all in a marketplace that never stops changing.

An "Aha!" Moment: Learning About Saccades

Luckily nature has been working on this fundamental challenging for a while. My breakthrough in thinking about this problem came from an unlikely source – the way our eyes work!

• Introducing Saccades: You might not be aware of it, but your eyes don't scan a scene smoothly. They make incredibly rapid, jerky movements called saccades.
• Fixations: Between these "jumps," your eyes pause briefly in "fixations" to gather visual information.
• The Brain's Magic: Your brain then expertly stitches these snapshots together to give you a clear, stable perception of the world.

It’s a highly efficient way to gather necessary information without needing to process everything continuously.

The Connection: Saccades & Our Dynamic Experimentation

This concept of "saccades" and "fixations" resonated perfectly with how we design our ad experiments:

• Fixation: We start by running an experiment for a few days (often initially at a low spending level). This gives us a baseline understanding and initial data on how different ad variants are performing.
• Saccade: Then, we introduce a clear, deliberate, and significant change. Often a decisive shift in budget allocation between the ad variants, and reshuffeling of which variants are randomized against each other. This is our purposeful "jump" to a new experimental state.
• Fixation: Then the next experiment runs for a few days. This fixation allows delayed performance measures to be considered (purchases aren't always immediate).
• Repeat: The change is quick (and coordinated) and then we wait for information to come in. It allows us to move and act, while blurring the picture as little as possible.

Sharper Focus, Faster Insights

Understanding saccades provided a fantastic mental model for why our dynamic approach works so well:

• Strategic "Jumps": Like our eyes, we don't just passively observe. We make strategic "jumps" (busts) to understand clearer cause-and-effect signals.
• Efficient Information Gathering: This method allows us to learn more about the best ads more effectively than just letting a test run for an extended period without intervention.
• Actionable Results for Clients: Ultimately, this means we provide our clients with more confident, actionable insights, and we can often do it faster and with a higher performing test.

It’s amazing how a little insight from biology can illuminate a statistical strategy! By mimicking our eyes' efficient way of seeing, we get a clearer view of what truly makes ads work.