2023
Working Paper
HBS Working Paper Series

Nailing Prediction: Experimental Evidence on the Value of Tools in Predictive Model Development

By: Daniel Yue, Paul Hamilton and Iavor Bojinov

Language:English
| Pages:75

Abstract

Predictive model development is understudied despite its centrality in modern artificial intelligence and machine learning business applications. Although prior discussions highlight advances in methods (along the dimensions of data, computing power, and algorithms) as the primary driver of model quality, the tools that implement those methods have been neglected. In a field experiment leveraging a predictive data science contest, we study the impact of tools by restricting access to software libraries for machine learning models. By only allowing access to these libraries in our control group, we find that teams with unrestricted access perform 30% better in log-loss error — a statistically and economically significant amount, equivalent to a 10-fold increase in the training data set size. We further find that teams with high general data-science skills are less affected by the intervention. In contrast, teams with high tool-specific skills significantly benefit from access to modeling libraries. Our findings are consistent with a mechanism we call ‘Tools-as-Skill,’ where tools automate and abstract some general data science skills but, in doing so, create the need for new tool-specific skills.