More from the Author

• Technical Note | HBS Case Collection | August 2020 (Revised September 2020)

  Assessing Prediction Accuracy of Machine Learning Models

  Michael W. Toffel, Natalie Epstein, Kris Ferreira and Yael Grushka-Cockayne

  The note introduces a variety of methods to assess the accuracy of machine learning prediction models. The note begins by briefly introducing machine learning, overfitting, training versus test datasets, and cross validation. The following accuracy metrics and tools are then described: mean squared error (MSE), mean absolute deviation (MAD), Brier score, and cross-entropy, true/false positives/negatives, the confusion matrix, true positive rate (sensitivity or recall), false negative rate (Type II error rate), precision, true negative rate (specificity), false positive rate (Type I error rate), receiver operating characteristic curve (ROC) and area under the curve (AUC), and precision-recall curve.

  Keywords: Machine learning; statistics; econometric analyses; experimental methods; data analysis; Data Analytics; Forecasting and Prediction; Data and Data Sets; Analysis; Mathematical Methods;

  Citation:

  Toffel, Michael W., Natalie Epstein, Kris Ferreira, and Yael Grushka-Cockayne. "Assessing Prediction Accuracy of Machine Learning Models." Harvard Business School Technical Note 621-045, August 2020. (Revised September 2020.)  View Details

  CiteView DetailsEducators

  Related

• Working Paper | 2020

  Market Segmentation Trees

  Ali Aouad, Adam Elmachtoub, Kris Ferreira and Ryan McNellis

  As more commerce and media consumption are being conducted online, a wealth of new opportunities are emerging for personalized advertising. We propose a general methodology, Model Trees for Personalization (MTP), for tackling a broad class of personalized decision-making problems including personalized advertising. The MTPs learn an interpretable market segmentation driven by differences in user behavior. Using this methodology, we design two new algorithms for fundamental problems in personalized advertising – Choice Model Trees (CMTs) for the user ad-response prediction problem, and Isotonic Regression Model Trees (IRMTs) for the bid landscape forecasting problem. We provide a customizable, computationally-efficient, and open-source code base for training MTPs in Python. We train our IRMT algorithm on historical bidding data from three different ad exchanges and show that the IRMT achieves 5-29% improvement in bid landscape forecasting accuracy over the model that a leading demand-side platform (DSP) provider currently uses in production.

  Keywords: Decision Trees; computational advertising; Market segmentation; Online Advertising; Customization and Personalization; Segmentation; Decision Making;

  Citation:

  Aouad, Ali, Adam Elmachtoub, Kris Ferreira, and Ryan McNellis. "Market Segmentation Trees." Working Paper, January 2020.  View Details

  CiteView Details Read Now

  Related

• Article | Journal of Marketing Research (JMR) | December 2019

  The Impact of Increasing Search Frictions on Online Shopping Behavior: Evidence from a Field Experiment

  Donald Ngwe, Kris J. Ferreira and Thales Teixeira

  Many online stores are designed such that shoppers can easily access any available discounted products. We propose that deliberately increasing search frictions by placing small obstacles to locating discounted items can improve online retailers’ margins and even increase conversion. We demonstrate using a simple theoretical framework that inducing consumers to inspect higher-priced items first may simultaneously increase the average price of items sold and the overall expected purchase probability by inducing consumers to search more products. We test and confirm these predictions in a series of field experiments conducted with a dominant online fashion and apparel retailer. Furthermore, using information in historical transaction data about each consumer, we demonstrate that price-sensitive shoppers are more likely to incur search costs to locate discounted items. Our results show that increasing search frictions can be used as a self-selecting price discrimination tool to match high discounts with price-sensitive consumers and full-priced offerings with price-insensitive consumers.

  Keywords: e-commerce; online retailing; friction; effor; search costs; price discrimination; Marketing; Consumer Behavior; Strategy; Price; Retail Industry; Fashion Industry;

  Citation:

  Ngwe, Donald, Kris J. Ferreira, and Thales Teixeira. "The Impact of Increasing Search Frictions on Online Shopping Behavior: Evidence from a Field Experiment." Journal of Marketing Research (JMR) 56, no. 6 (December 2019): 944–959.  View Details

  CiteView DetailsFind at Harvard

  Related