Skip to Main Content
HBS Home
  • About
  • Academic Programs
  • Alumni
  • Faculty & Research
  • Baker Library
  • Giving
  • Harvard Business Review
  • Initiatives
  • News
  • Recruit
  • Map / Directions
Faculty & Research
  • Faculty
  • Research
  • Featured Topics
  • Academic Units
  • …→
  • Harvard Business School→
  • Faculty & Research→
Publications
Publications
  • September–October 2017
  • Article
  • Harvard Business Review

The Surprising Power of Online Experiments: Getting the Most Out of A/B and Other Controlled Tests

By: Ron Kohavi and Stefan Thomke
  • Format:Print
ShareBar

Abstract

In the fast-moving digital world, even experts have a hard time assessing new ideas. Case in point: At Bing, a small headline change an employee proposed was deemed a low priority and shelved for months until one engineer decided to do a quick online controlled experiment—an A/B test—to try it out. The test showed that the change increased revenue by an astonishing 12%. It ended up being the best revenue-generating idea Bing ever had, worth $100 million. That experience illustrates why it’s critical to adopt an “experiment with everything” approach, say Kohavi, the head of the Analysis & Experimentation team at Microsoft, and Thomke, an HBS professor. In this article they describe how to properly design and execute A/B and other controlled tests, ensure their integrity, interpret results, and avoid pitfalls. They argue that if a company sets up the right infrastructure and software, it will be able to evaluate ideas not only for improving websites but also for new business models, products, strategies, and marketing campaigns—all relatively inexpensively. This will help it find the right path forward, especially when answers aren’t obvious or people have conflicting opinions.

Keywords

Experiments; A/b Testing; Research; Consumer Behavior

Citation

Kohavi, Ron, and Stefan Thomke. "The Surprising Power of Online Experiments: Getting the Most Out of A/B and Other Controlled Tests." Harvard Business Review 95, no. 5 (September–October 2017): 74–82.
  • Find it at Harvard
  • Register to Read

About The Author

Stefan H. Thomke

Technology and Operations Management
→More Publications

More from the Authors

    • July 2020
    • Faculty Research

    Ferrari

    By: Stefan H. Thomke
    • July 2020
    • Faculty Research

    A. Lange & Söhne

    By: Stefan Thomke
    • June 2020
    • Faculty Research

    Sony

    By: Stefan H. Thomke
More from the Authors
  • Ferrari By: Stefan H. Thomke
  • A. Lange & Söhne By: Stefan Thomke
  • Sony By: Stefan H. Thomke
ǁ
Campus Map
Harvard Business School
Soldiers Field
Boston, MA 02163
→Map & Directions
→More Contact Information
  • Make a Gift
  • Site Map
  • Jobs
  • Harvard University
  • Trademarks
  • Policies
  • Digital Accessibility
Copyright © President & Fellows of Harvard College