The Power of Experiments

Michael  Luca; Max H. Bazerman

Book | 2020

The Power of Experiments: Decision-Making in a Data-Driven World

Abstract

Have you logged into Facebook recently? Searched for something on Google? Chosen a movie on Netflix? If so, you've probably been an unwitting participant in a variety of experiments—also known as randomized controlled trials—designed to test the impact of changes to an experience or product. Once an esoteric tool for academic research, the randomized controlled trial has gone mainstream—and is becoming an important part of the managerial toolkit. In The Power of Experiments: Decision-Making in a Data Driven World, Michael Luca and Max Bazerman explore the value of experiments and the ways in which they can improve organizational decisions. Drawing on real world experiments and case studies, Luca and Bazerman show that going by gut is no longer enough—successful leaders need frameworks for moving between data and decisions. Experiments can save companies money—eBay, for example, discovered how to cut $50 million from its yearly advertising budget without losing customers. Experiments can also bring to light something previously ignored, as when Airbnb was forced to confront rampant discrimination by its hosts. The Power of Experiments introduces readers to the topic of experimentation and the managerial challenges that surround them. Looking at experiments in the tech sector and beyond, this book offers lessons and best practices for making the most of experiments.

In The Power of Experiments: Decision-Making in a Data Driven World, Michael Luca and Max Bazerman explore the value of experiments, and the ways in which they can improve organizational decisions. Drawing on real world experiments and case studies, Luca and Bazerman show that going by gut is no longer enough—successful leaders need frameworks for moving between data and decisions. Experiments can save companies money—eBay, for example, discovered how to cut $50 million from its yearly advertising budget without losing customers. Experiments can also bring to light something previously ignored, as when Airbnb was forced to confront rampant discrimination by its hosts.

The Power of Experiments introduces readers to the topic of experimentation and the managerial challenges that surround them. Looking at experiments in the tech sector and beyond, this book offers lessons and best practices for making the most of experiments.

Keywords: experiments; randomized controlled trials; Organizations; Decision Making; Data and Data Sets; Management Analysis, Tools, and Techniques;

Format: Print

Find at Harvard Purchase

Citation:

Luca, Michael, and Max H. Bazerman. The Power of Experiments: Decision-Making in a Data-Driven World. Cambridge, MA: MIT Press, 2020.

The Paycheck Protection Program (PPP) aimed to quickly deliver hundreds of billions of dollars of loans to small businesses, with the loans administered via private banks. In this paper, we use firm-level data to document the demand and supply of PPP funds. Using an instrumental variables approach, we find that PPP loans led to a 14 to 30 percentage point increase in a business’s expected survival, and a positive but imprecise effect on employment. Moreover, the effects on survival were heterogeneous and highlight an important tradeoff faced by policymakers: while administering the loans via private banks allowed for rapid delivery of funds, it also limited the government’s ability to target the funding—instead allowing pre-existing connections between businesses and banks to determine which firms would benefit from the program.

During the COVID-19 pandemic, states issued and then rescinded stay-at-home orders that restricted mobility. We develop a model of learning by deregulation, which predicts that lifting stay-at-home orders can signal that going out has become safer. Using restaurant activity data, we find that the implementation of stay-at-home orders initially had a limited impact, but that activity rose quickly after states’ reopenings. The results suggest that consumers inferred from reopening that it was safer to eat out. The rational, but mistaken inference that occurs in our model may explain why a sharp rise of COVID-19 cases followed reopening in some states.

To explore the impact of COVID on small businesses, we conducted a survey of more than 5,800 small businesses between March 28 and April 4, 2020. Several themes emerged. First, mass layoffs and closures had already occurred—just a few weeks into the crisis. Second, the risk of closure was negatively associated with the expected length of the crisis. Moreover, businesses had widely varying beliefs about the likely duration of COVID-related disruptions. Third, many small businesses are financially fragile: the median business with more than $10,000 in monthly expenses had only about two weeks of cash on hand at the time of the survey. Fourth, the majority of businesses planned to seek funding through the CARES act. However, many anticipated problems with accessing the program, such as bureaucratic hassles and difficulties establishing eligibility. Using experimental variation, we also assess take-up rates and business resilience effects for loans relative to grants-based programs.

Faculty & Research

The Power of Experiments: Decision-Making in a Data-Driven World

Abstract

Citation:

About the Authors

More from these Authors