Skip to Main Content
HBS Home
  • About
  • Academic Programs
  • Alumni
  • Faculty & Research
  • Baker Library
  • Giving
  • Harvard Business Review
  • Initiatives
  • News
  • Recruit
  • Map / Directions
Faculty & Research
  • Faculty
  • Research
  • Featured Topics
  • Academic Units
  • …→
  • Harvard Business School→
  • Faculty & Research→
Publications
Publications
  • February 2021
  • Tutorial

Assessing Prediction Accuracy of Machine Learning Models

By: Michael Toffel and Natalie Epstein
  • Format:Video
  • | Language:English
ShareBar

Abstract

This video describes how to assess the accuracy of machine learning prediction models, primarily in the context of machine learning models that predict binary outcomes, such as logistic regression, random forest, or nearest neighbor models. After introducing and differentiating the concepts of training and testing data, the video presents the confusion matrix and uses it to describe a series of accuracy metrics including true/false positives/negatives, true positive rate (sensitivity or recall), false negative rate (Type II error rate), precision, true negative rate (specificity), and false positive rate (Type I error rate). It also addresses the impact of setting thresholds to convert continuous predictions to binary classifications, and describes the receiver operating characteristic curve (ROC curve) and area under the curve (AUC). Several examples are provided, including one where during the middle of a course, an instructor is viewing students’ interim grades, attendance, and participation, and needs to decide which students should seek tutoring to avoid a poor final grade. This video can be assigned in conjunction with the “Assessing Prediction Accuracy of Machine Learning Models” technical note (HBS No. 621045).

Keywords

Machine Learning; Statistics; Experiments; Forecasting and Prediction; Performance Evaluation

Citation

Toffel, Michael, and Natalie Epstein. Assessing Prediction Accuracy of Machine Learning Models. Harvard Business School Tutorial 621-706, February 2021.
  • Purchase

About The Author

Michael W. Toffel

Technology and Operations Management
→More Publications

More from the Authors

    • January 2023
    • Faculty Research

    Global Sourcing at Nike

    By: Nien-hê Hsieh and Michael W. Toffel
    • December 2022
    • Faculty Research

    Driving Decarbonization at BMW - Spreadsheet Supplement

    By: Shirley Lu, George Serafeim and Michael W. Toffel
    • December 2022
    • Faculty Research

    Driving Decarbonization at BMW - Video Supplement

    By: Shirley Lu, George Serafeim and Michael W. Toffel
More from the Authors
  • Global Sourcing at Nike By: Nien-hê Hsieh and Michael W. Toffel
  • Driving Decarbonization at BMW - Spreadsheet Supplement By: Shirley Lu, George Serafeim and Michael W. Toffel
  • Driving Decarbonization at BMW - Video Supplement By: Shirley Lu, George Serafeim and Michael W. Toffel
ǁ
Campus Map
Harvard Business School
Soldiers Field
Boston, MA 02163
→Map & Directions
→More Contact Information
  • Make a Gift
  • Site Map
  • Jobs
  • Harvard University
  • Trademarks
  • Policies
  • Accessibility
  • Digital Accessibility
Copyright © President & Fellows of Harvard College