Skip to Main Content
HBS Home
  • About
  • Academic Programs
  • Alumni
  • Faculty & Research
  • Baker Library
  • Giving
  • Harvard Business Review
  • Initiatives
  • News
  • Recruit
  • Map / Directions
Faculty & Research
  • Faculty
  • Research
  • Featured Topics
  • Academic Units
  • …→
  • Harvard Business School→
  • Faculty & Research→
Publications
Publications
  • 2023
  • Article
  • Journal of the American Statistical Association

Experimental Evaluation of Individualized Treatment Rules

By: Kosuke Imai and Michael Lingzhi Li
  • Format:Print
  • | Pages:15
ShareBar

Abstract

The increasing availability of individual-level data has led to numerous applications of individualized (or personalized) treatment rules (ITRs). Policy makers often wish to empirically evaluate ITRs and compare their relative performance before implementing them in a target population. We propose a new evaluation metric, the population average prescriptive effect (PAPE). The PAPE compares the performance of ITR with that of non-individualized treatment rule, which randomly treats the same proportion of units. Averaging the PAPE over a range of budget constraints yields our second evaluation metric, the area under the prescriptive effect curve (AUPEC). The AUPEC represents an overall performance measure for evaluation, like the area under the receiver and operating characteristic curve (AUROC) does for classification, and is a generalization of the QINI coefficient used in uplift modeling. We use Neyman’s repeated sampling framework to estimate the PAPE and AUPEC and derive their exact finite-sample variances based on random sampling of units and random assignment of treatment. We extend our methodology to a common setting, in which the same experimental data are used to both estimate and evaluate ITRs. In this case, our variance calculation incorporates the additional uncertainty due to random splits of data used for cross-validation. The proposed evaluation metrics can be estimated without requiring modeling assumptions, asymptotic approximation, or resampling methods. As a result, it is applicable to any ITR including those based on complex machine learning algorithms. The open-source software package is available for implementing the proposed methodology. Supplementary materials for this article are available online.

Keywords

Causal Inference; Heterogeneous Treatment Effects; Precision Medicine; Uplift Modeling; Analytics and Data Science; AI and Machine Learning

Citation

Imai, Kosuke, and Michael Lingzhi Li. "Experimental Evaluation of Individualized Treatment Rules." Journal of the American Statistical Association 118, no. 541 (2023): 242–256.
  • Find it at Harvard
  • Read Now

About The Author

Michael Lingzhi Li

Technology and Operations Management
→More Publications

More from the Authors

    • 2025
    • Journal of Business & Economic Statistics

    Statistical Inference for Heterogeneous Treatment Effects Discovered by Generic Machine Learning in Randomized Experiments

    By: Kosuke Imai and Michael Lingzhi Li
    • 2024
    • Journal of Causal Inference

    Neyman Meets Causal Machine Learning: Experimental Evaluation of Individualized Treatment Rules

    By: Michael Lingzhi Li and Kosuke Imai
    • 2024
    • Faculty Research

    Learning to Cover: Online Learning and Optimization with Irreversible Decisions

    By: Alexander Jacquillat and Michael Lingzhi Li
More from the Authors
  • Statistical Inference for Heterogeneous Treatment Effects Discovered by Generic Machine Learning in Randomized Experiments By: Kosuke Imai and Michael Lingzhi Li
  • Neyman Meets Causal Machine Learning: Experimental Evaluation of Individualized Treatment Rules By: Michael Lingzhi Li and Kosuke Imai
  • Learning to Cover: Online Learning and Optimization with Irreversible Decisions By: Alexander Jacquillat and Michael Lingzhi Li
ǁ
Campus Map
Harvard Business School
Soldiers Field
Boston, MA 02163
→Map & Directions
→More Contact Information
  • Make a Gift
  • Site Map
  • Jobs
  • Harvard University
  • Trademarks
  • Policies
  • Accessibility
  • Digital Accessibility
Copyright © President & Fellows of Harvard College.