More from these Authors

• Technical Note | HBS Case Collection | August 2020 (Revised September 2020)

  Assessing Prediction Accuracy of Machine Learning Models

  Michael W. Toffel, Natalie Epstein, Kris Ferreira and Yael Grushka-Cockayne

  The note introduces a variety of methods to assess the accuracy of machine learning prediction models. The note begins by briefly introducing machine learning, overfitting, training versus test datasets, and cross validation. The following accuracy metrics and tools are then described: mean squared error (MSE), mean absolute deviation (MAD), Brier score, and cross-entropy, true/false positives/negatives, the confusion matrix, true positive rate (sensitivity or recall), false negative rate (Type II error rate), precision, true negative rate (specificity), false positive rate (Type I error rate), receiver operating characteristic curve (ROC) and area under the curve (AUC), and precision-recall curve.

  Keywords: Machine learning; statistics; econometric analyses; experimental methods; data analysis; Data Analytics;

  Citation:

  Toffel, Michael W., Natalie Epstein, Kris Ferreira, and Yael Grushka-Cockayne. "Assessing Prediction Accuracy of Machine Learning Models." Harvard Business School Technical Note 621-045, August 2020. (Revised September 2020.)  View Details

  CiteView DetailsEducators

  Related

• Case | HBS Case Collection | September 2020

  Student Success at Georgia State University (A)

  Michael W. Toffel, Robin Mendelson and Julia Kelley

  Georgia State University had developed a reputation for driving student success by nearly doubling its graduation rate for students of all racial, ethnic, and socioeconomic backgrounds. It did so while growing its student body and the proportion of Black/African American, low-income, and first-generation students—groups with historically lower postsecondary graduation rates compared to national averages. Georgia State’s Student Success team, led by Tim Renick and Allison Calhoun-Brown, used a data-based approach to deploy micro-grant programs to retain students, implemented predictive analytics to improve student advising, and optimized course sequencing to help students graduate before they exhausted their financial aid. In 2016, they faced a growing “summer melt” problem where nearly 20% of incoming students who committed to attend never actually enrolled at Georgia State—and many never enrolled at any college. At the same time, they wondered how to balance continuing to incrementally improve student success at Georgia State and scaling their efforts to help the many other universities facing similar problems who sought to learn from their experience.

  Keywords: Education; Higher Education; Learning; Curriculum and Courses; Demographics; Diversity; Ethnicity; Income; Race; Leadership; Goals and Objectives; Measurement and Metrics; Operations; Organizations; Mission and Purpose; Organizational Culture; Outcome or Result; Performance; Performance Effectiveness; Performance Evaluation; Performance Improvement; Planning; Strategic Planning; Social Enterprise; Nonprofit Organizations; Social Issues; Wealth and Poverty; Equality and Inequality; Technology; Technology Platform; Education Industry; Atlanta;

  Citation:

  Toffel, Michael W., Robin Mendelson, and Julia Kelley. "Student Success at Georgia State University (A)." Harvard Business School Case 621-006, September 2020.  View Details

  CiteView DetailsEducators

  Related

• Supplement | HBS Case Collection | September 2020

  Student Success at Georgia State University (B)

  Michael W. Toffel, Robin Mendelson and Julia Kelley

  This is a supplement to the Student Success at Georgia State University (A) case. The (B) case includes the results of a randomized control trial that Georgia State conducted to test education technology start-up AdmitHub’s chatbot solution as a strategy for improving “summer melt” (i.e., applicants confirming that they planned to enroll but then not enrolling).

  Keywords: Education; Higher Education; Learning; Curriculum and Courses; Demographics; Diversity; Ethnicity; Income; Race; Values and Beliefs; Leadership; Goals and Objectives; Measurement and Metrics; Operations; Organizations; Mission and Purpose; Organizational Culture; Outcome or Result; Performance; Performance Effectiveness; Performance Evaluation; Performance Improvement; Planning; Strategic Planning; Social Enterprise; Nonprofit Organizations; Social Issues; Wealth and Poverty; Equality and Inequality; Technology; Technology Platform; Education Industry; Atlanta;

  Citation:

  Toffel, Michael W., Robin Mendelson, and Julia Kelley. "Student Success at Georgia State University (B)." Harvard Business School Supplement 621-039, September 2020.  View Details

  CiteView Details

  Related