More from the Author

• Technical Note | HBS Case Collection | August 2020 (Revised September 2020)

  Assessing Prediction Accuracy of Machine Learning Models

  Michael W. Toffel, Natalie Epstein, Kris Ferreira and Yael Grushka-Cockayne

  The note introduces a variety of methods to assess the accuracy of machine learning prediction models. The note begins by briefly introducing machine learning, overfitting, training versus test datasets, and cross validation. The following accuracy metrics and tools are then described: mean squared error (MSE), mean absolute deviation (MAD), Brier score, and cross-entropy, true/false positives/negatives, the confusion matrix, true positive rate (sensitivity or recall), false negative rate (Type II error rate), precision, true negative rate (specificity), false positive rate (Type I error rate), receiver operating characteristic curve (ROC) and area under the curve (AUC), and precision-recall curve.

  Keywords: Machine learning; statistics; econometric analyses; experimental methods; data analysis; Data Analytics; Forecasting and Prediction; Data and Data Sets; Analysis; Mathematical Methods;

  Citation:

  Toffel, Michael W., Natalie Epstein, Kris Ferreira, and Yael Grushka-Cockayne. "Assessing Prediction Accuracy of Machine Learning Models." Harvard Business School Technical Note 621-045, August 2020. (Revised September 2020.)  View Details

  CiteView DetailsEducators

  Related

• Exercise | September 2013 (Revised September 2020)

  Process Analytics Simulation: Solutions

  Robert S. Huckman, Aizan Radzi, Willy Shih, Roy D. Shapiro and Michael W. Toffel

  A set of exercises and instructions to be used with the Process Simulator software made by Pro Model. These exercises allow students to investigate the impact of variable processing times on the performance of simple in-line processes. Includes color exhibits.

  Keywords: Operations; Business Processes; Performance; Measurement and Metrics;

  Citation:

  Huckman, Robert S., Aizan Radzi, Willy Shih, Roy D. Shapiro, and Michael W. Toffel. "Process Analytics Simulation: Solutions." Harvard Business School Exercise 614-029, September 2013. (Revised September 2020.)  View Details

  CiteView DetailsPurchase

  Related

• Case | HBS Case Collection | September 2020

  Student Success at Georgia State University (A)

  Michael W. Toffel, Robin Mendelson and Julia Kelley

  Georgia State University had developed a reputation for driving student success by nearly doubling its graduation rate for students of all racial, ethnic, and socioeconomic backgrounds. It did so while growing its student body and the proportion of Black/African American, low-income, and first-generation students—groups with historically lower postsecondary graduation rates compared to national averages. Georgia State’s Student Success team, led by Tim Renick and Allison Calhoun-Brown, used a data-based approach to deploy micro-grant programs to retain students, implemented predictive analytics to improve student advising, and optimized course sequencing to help students graduate before they exhausted their financial aid. In 2016, they faced a growing “summer melt” problem where nearly 20% of incoming students who committed to attend never actually enrolled at Georgia State—and many never enrolled at any college. At the same time, they wondered how to balance continuing to incrementally improve student success at Georgia State and scaling their efforts to help the many other universities facing similar problems who sought to learn from their experience.

  Keywords: Education; Higher Education; Learning; Curriculum and Courses; Demographics; Diversity; Ethnicity; Income; Race; Leadership; Goals and Objectives; Measurement and Metrics; Operations; Organizations; Mission and Purpose; Organizational Culture; Outcome or Result; Performance; Performance Effectiveness; Performance Evaluation; Service Operations; Performance Improvement; Planning; Strategic Planning; Social Enterprise; Nonprofit Organizations; Social Issues; Wealth and Poverty; Equality and Inequality; Technology; Technology Platform; Education Industry; Atlanta;

  Citation:

  Toffel, Michael W., Robin Mendelson, and Julia Kelley. "Student Success at Georgia State University (A)." Harvard Business School Case 621-006, September 2020.  View Details

  CiteView DetailsEducators

  Related