Skip to Main Content
HBS Home
  • About
  • Academic Programs
  • Alumni
  • Faculty & Research
  • Baker Library
  • Giving
  • Harvard Business Review
  • Initiatives
  • News
  • Recruit
  • Map / Directions
Faculty & Research
  • Faculty
  • Research
  • Featured Topics
  • Academic Units
  • …→
  • Harvard Business School→
  • Faculty & Research→
Publications
Publications
  • June 23, 2021
  • Article
  • Harvard Business Review Digital Articles

Research: When A/B Testing Doesn't Tell You the Whole Story

By: Eva Ascarza
  • Format:Electronic
ShareBar

Abstract

When it comes to churn prevention, marketers traditionally start by identifying which customers are most likely to churn, and then running A/B tests to determine whether a proposed retention intervention will be effective at retaining those high-risk customers. While this strategy can be effective, the author shares new research based on field experiments with over 14,000 customers that suggests it isn’t always the best way to maximize ROI on marketing spend. Instead, the author argues that firms should use A/B test data alongside customers’ behavioral and demographic data to determine which subgroup of customers will be most sensitive to the specific intervention that’s being considered. Importantly, the data suggests that this subgroup doesn’t necessarily correspond to the “high-risk” customer group—in other words, it’s very possible that the intervention won’t be as effective at retaining high-risk customers as it will be at retaining some other group of customers. By identifying the characteristics that actually correlate with high sensitivity to a given intervention, marketers can proactively target their campaigns at the customers who will be most receptive to them, ultimately reducing churn rates and increasing ROI.

Keywords

Customer Retention; Churn; Targeting; Market Research; Marketing; Investment Return; Customers; Retention; Research

Citation

Ascarza, Eva. "Research: When A/B Testing Doesn't Tell You the Whole Story." Harvard Business Review Digital Articles (June 23, 2021).
  • Find it at Harvard
  • Register to Read

About The Author

Eva Ascarza

Marketing
→More Publications

More from the Author

    • 2022
    • Faculty Research

    When Less Is More: Using Short-term Signals to Overcome Systematic Bias in Long-run Targeting

    By: Ta-Wei Huang and Eva Ascarza
    • August 2022
    • Faculty Research

    Retail Media Networks

    By: Eva Ascarza, Ayelet Israeli and Celine Chammas
    • March 2022 (Revised March 2022)
    • Faculty Research

    Managing Customers in the Digital Era

    By: Eva Ascarza
More from the Author
  • When Less Is More: Using Short-term Signals to Overcome Systematic Bias in Long-run Targeting By: Ta-Wei Huang and Eva Ascarza
  • Retail Media Networks By: Eva Ascarza, Ayelet Israeli and Celine Chammas
  • Managing Customers in the Digital Era By: Eva Ascarza
ǁ
Campus Map
Harvard Business School
Soldiers Field
Boston, MA 02163
→Map & Directions
→More Contact Information
  • Make a Gift
  • Site Map
  • Jobs
  • Harvard University
  • Trademarks
  • Policies
  • Accessibility
  • Digital Accessibility
Copyright © President & Fellows of Harvard College