From Data to Decisions: The Role of Experiments - Harvard Business School MBA Program

From Data to Decisions: The Role of Experiments

Course Number 2205

Associate Professor Michael Luca
Spring; Q4; 1.5 credits
14 sessions
Final project

Overview

Experimental methods. Control groups. Human subjects. The very words conjure up images of mad scientists concocting their latest schemes. But experiments are no longer confined to the esoteric world of academic researchers. Over the past decade, experiments have gone mainstream.

Experiments have become particularly central to the ethos of decision-making in tech companies, with many companies running thousands of experiments per year. Tinder runs experiments to figure out how to increase the probability users will find a good match. Uber runs experiments to decide everything from how much to pay drivers to whether or not to require riders to leave reviews. Through a series of experiments, eBay realized that it had been wasting millions on Google advertisements.

Experiments extend well beyond the tech world as well. A recent count shows that out of the 50 largest U.S. companies, at least 45 have run experiments. Marketers run experiments to understand which advertisements work, and which are ineffective. Governments run experiments to test ideas and evaluate policies. Human resource and people analytics departments are increasingly using experiments to know which incentive systems are most effective. This course will explore various ways in which experiments can yield insights for organizations — in areas such as platform design, strategy, operations, human resources, and marketing.

The overarching goal of this course is to help students develop an experimental mindset, and an appreciation of how experiments within organizations can be used to improve managerial decision-making.  This course will help to familiarize students with the fundamentals of randomized controlled trials, allowing them to get the most from experimental insights and to work effectively in an increasingly data-driven world.

Course Objectives

This course has four objectives:
  • First, to demystify and clarify the experimental method, which is often cast in alarmist tones (consider this headline from the Atlantic: Everything We Know About Facebook's Secret Mood Manipulation).
  • Second, to shed light on the types of insight that can be gained from experiments, and where and when experiments are most useful. Through exploring prior experiments run within organizations, the course will help students to approach managerial problems with an experimental mindset.
  • Third, to develop frameworks for interpreting, designing, and analyzing experiments, and provide students with a basic understanding of experimental methods.
  • Fourth, to develop frameworks for understanding of the strengths and limitations of experiments, allowing students to develop the frameworks to systematically use of experiments within an organization, and to avoid misuse of data and experiments.

With these goals in mind, the course will consist of a combination of case studies, empirical exercises (including analyzing data sets), readings, and guests. Tentative guests include managers and data scientists from organizations ranging from Uber and Yelp to government agencies. The course will involve a final project. No technical background is required.