The Surprising Power of Online Experiments: Getting the Most Out of A/B and Other Controlled Tests

In the fast-moving digital world, even experts have a hard time assessing new ideas. Case in point: At Bing, a small headline change an employee proposed was deemed a low priority and shelved for months until one engineer decided to do a quick online controlled experiment—an A/B test—to try it out. The test showed that the change increased revenue by an astonishing 12%. It ended up being the best revenue-generating idea Bing ever had, worth $100 million. That experience illustrates why it’s critical to adopt an “experiment with everything” approach, say Kohavi, the head of the Analysis & Experimentation team at Microsoft, and Thomke, an HBS professor. In this article they describe how to properly design and execute A/B and other controlled tests, ensure their integrity, interpret results, and avoid pitfalls. They argue that if a company sets up the right infrastructure and software, it will be able to evaluate ideas not only for improving websites but also for new business models, products, strategies, and marketing campaigns—all relatively inexpensively. This will help it find the right path forward, especially when answers aren’t obvious or people have conflicting opinions.