O'Reilly logo

A/B Testing: The Most Powerful Way to Turn Clicks Into Customers by Pete Koomen, Dan Siroker

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

CHAPTER10

Iterate, Iterate, Iterate

The Art of Asking Many Small Questions Rather than One Big One

When test results first start to come in, it's important to recognize that they're likely going to generate more questions than answers. These questions will likely also point to more assumptions you find you want to test. That's okay; in fact, that's how it's supposed to work. And that's the perfect starting point for your tests to come.

One of the biggest questions that we had when starting Optimizely was whether people would continue experimenting after they found a “local maximum,” that is, the best set of tweaks for their current design or current funnel. We worried that people would spend a few months optimizing a site—a better headline here, different image there—and be done with it. We didn't want businesses to think of A/B testing as a finite, onetime process, and we acknowledged the disconcerting possibility that businesses would feel satisfied if the site worked better than before, and might throw in the towel and declare, “Well, enough of that! We're all done A/B testing.”

Much to our delight, we've found that the opposite has occurred. After a few big wins, most people realize that they're just beginning the testing journey—and the success they've enjoyed propels them to keep going.

In month one, you run a couple of tests; you get some wins and you're adding those wins into the production code. Those wins might have been scattered around different pages of your site, ...

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required