Skip to Main Content
HBS Home
  • About
  • Academic Programs
  • Alumni
  • Faculty & Research
  • Baker Library
  • Giving
  • Harvard Business Review
  • Initiatives
  • News
  • Recruit
  • Map / Directions
Faculty & Research
  • Faculty
  • Research
  • Featured Topics
  • Academic Units
  • …→
  • Harvard Business School→
  • Faculty & Research→
Publications
Publications
  • Forthcoming
  • Article
  • Management Science

Design and Analysis of Switchback Experiments

By: Iavor I Bojinov, David Simchi-Levi and Jinglong Zhao
  • Format:Print
  • | Pages:19
ShareBar

Abstract

In switchback experiments, a firm sequentially exposes an experimental unit to a random treatment, measures its response, and repeats the procedure for several periods to determine which treatment leads to the best outcome. Although practitioners have widely adopted this experimental design technique, the development of its theoretical properties and the derivation of optimal design procedures have been, to the best of our knowledge, elusive. In this paper, we address these limitations by establishing the necessary results to ensure that practitioners can apply this powerful class of experiments with minimal assumptions. Our main result is the derivation of the optimal design of switchback experiments under a range of different assumptions on the order of carryover effect—that is, the length of time a treatment persists in impacting the outcome. We cast the experimental design problem as a minimax discrete robust optimization problem, identify the worst-case adversarial strategy, establish structural results for the optimal design, and finally solve the problem via a continuous relaxation. For the optimal design, we derive two approaches for performing inference after running the experiment. The first provides exact randomization based p-values and the second uses a finite population central limit theorem to conduct conservative hypothesis tests and build confidence intervals. We further provide theoretical results for our inferential procedures when the order of the carryover effect is misspecified. For firms that possess the capability to run multiple switchback experiments, we also provide a data-driven strategy to identify the likely order of carryover effect. To study the empirical properties of our results, we conduct extensive simulations. We conclude the paper by providing some practical suggestions.

Keywords

Switchback Experiments; Design; Analysis; Mathematical Methods

Citation

Bojinov, Iavor I., David Simchi-Levi, and Jinglong Zhao. "Design and Analysis of Switchback Experiments." Management Science (forthcoming). (Pre-published online November 1, 2022.)
  • Find it at Harvard
  • Read Now

About The Author

Iavor I. Bojinov

Technology and Operations Management
→More Publications

More from the Authors

    • 2022
    • Faculty Research

    Nailing Prediction: Experimental Evidence on the Value of Tools in Predictive Model Development

    By: Daniel Yue, Paul Hamilton and Iavor Bojinov
    • October 2022
    • Faculty Research

    On Ramp to Crypto

    By: Iavor Bojinov, Michael Parzen and Paul Hamilton
    • September 16, 2022
    • Science

    A Causal Test of the Strength of Weak Ties

    By: Karthik Rajkumar, Guillaume Saint-Jacques, Iavor I. Bojinov, Erik Brynjolfsson and Sinan Aral
More from the Authors
  • Nailing Prediction: Experimental Evidence on the Value of Tools in Predictive Model Development By: Daniel Yue, Paul Hamilton and Iavor Bojinov
  • On Ramp to Crypto By: Iavor Bojinov, Michael Parzen and Paul Hamilton
  • A Causal Test of the Strength of Weak Ties By: Karthik Rajkumar, Guillaume Saint-Jacques, Iavor I. Bojinov, Erik Brynjolfsson and Sinan Aral
ǁ
Campus Map
Harvard Business School
Soldiers Field
Boston, MA 02163
→Map & Directions
→More Contact Information
  • Make a Gift
  • Site Map
  • Jobs
  • Harvard University
  • Trademarks
  • Policies
  • Accessibility
  • Digital Accessibility
Copyright © President & Fellows of Harvard College