Course Overview

Data and experiments are increasingly used to guide managerial decision-making. Large tech companies run thousands of experiments per year to test everything from product design to pricing to which markets to enter and how. Human resource and people analytics departments use experiments to understand which incentive systems are most effective. Marketers run experiments to understand which advertisements increase sales. Governments run experiments to evaluate policies in areas ranging from education to encouraging entrepreneurship. More broadly, industries ranging from consulting to tech depend on systematic data analysis to complement managerial judgment.

Effective strategy and managerial judgment in today’s world requires a foundational understanding of data and experiments, which is a core component of the modern analytics toolkit. The overarching goal of the course is to help students gain effective tools and leverage data and experimental insights to inform managerial and strategic decisions in an increasingly complex and data-driven world.

Course Structure

The course has three modules:

Module 1: Finding signal in the noise

The world is increasingly filled with data, and we are regularly bombarded with facts and figures. Being a good consumer of data analytics starts with gaining a basic understanding of data. Drawing on research from behavioral economics and applied econometrics, this module will introduce MBA students to a set of common mistakes people make when interpreting data, and how to avoid these mistakes. It will help them be good consumers of data, and to diagnose whether or not a fact is compelling. Students will learn to recognize common challenges of observational data. The module will also introduce students to being producers of basic visualizations and summaries of data, allowing them to transparently communicate using data. The goal is to help students become more fluent in thinking about data.

Module 2: Leveraging experiments

Armed with a basic fluency in data, module 2 will help students to develop a toolkit around a powerful approach to identifying causal relationships – experiments. This module will help students gain an appreciation of the value of experimentation. It will provide a foundational understanding of how to interpret and design experiments in an organization. And, it will more broadly help them to approach questions with a causal lens.

This module focuses on three broad questions:

• When should organizations run experiments? Through case studies, the course will help students to approach managerial problems with an experimental mindset, and shed light on the types of insight that can be gained from experiments, as well as the disasters that can be avoided through careful experimental analysis.
• What are the core concepts involved in designing and interpreting an experiment? The course will develop frameworks for interpreting, designing, and analyzing experiments, provide a solid understanding of experimental methods, and offer insights on how to identify the mistakes even smart leaders and managers make when they try to gather and interpret data that will guide their strategic decisions.
• How should managers incorporate experimental evidence into managerial decisions? This course will develop frameworks for students to better understand and address the challenges that arise in using experiments to inform managerial judgment. Examining real-world experiments that organizations have run, students will gain an understanding of how experimental evidence fits into the broader picture and how managers can move from experimental evidence to decisions.

Module 3: Creating a culture of data driven decision making

This module will look at the broader role of data and experiments within organizations. It will go through cases of organizations - ranging from Airbnb and StubHub to the UK tax department to the National Basketball Association - that made the transition to taking a more data driven approach. This module will explore lessons that these and organizations learned along the way, and develop a toolkit for creating a culture of data-driven decisions.

Course format and focus

The focus will draw heavily on the tech sector (which has been an early adopter of data analytics). However, the lessons will apply more broadly, and the course will explore a variety of contexts, such as platform design, strategy, operations, human resources, and marketing. The course will consist of a combination of case studies, book chapters and articles, as well as guest discussions and panels. The course will involve a final project. No technical background is required.

For more discussion of the content of the types of topics covered in the course, please see this article: https://hbswk.hbs.edu/item/steer-clear-of-the-blind-spots-that-derail-experiments

Copyright © 2021 President & Fellows of Harvard College. All Rights Reserved.