Skip to Main Content
HBS Home
  • About
  • Academic Programs
  • Alumni
  • Faculty & Research
  • Baker Library
  • Giving
  • Harvard Business Review
  • Initiatives
  • News
  • Recruit
  • Map / Directions
Faculty & Research
  • Faculty
  • Research
  • Featured Topics
  • Academic Units
  • …→
  • Harvard Business School→
  • Faculty & Research→
Publications
Publications
  • 2022
  • Working Paper
  • HBS Working Paper Series

Nailing Prediction: Experimental Evidence on the Value of Tools in Predictive Model Development

By: Daniel Yue, Paul Hamilton and Iavor Bojinov
  • | Pages:74
ShareBar

Abstract

Predictive model development is understudied despite its importance to modern businesses. Although prior discussions highlight advances in methods (along the dimensions of data, computing power, and algorithms) as the primary driver of model quality, the value of tools that implement those methods has been neglected. In a field experiment leveraging a predictive data science contest, we study the importance of tools by restricting access to software libraries for machine learning models. By only allowing access to these libraries in our control group, we find that teams with unrestricted access perform 30% better in log-loss error — a statistically and economically significant amount, equivalent to a 10-fold increase in the training data set size. We further find that teams with high general data-science skills are less affected by the intervention, while teams with high tool-specific skills significantly benefit from access to modeling libraries. Our findings are consistent with a mechanism we call ‘Tools-as-Skill,’ where tooling automates and abstracts some general data science skills but, in doing so, creates the need for new tool-specific skills.

Keywords

Analytics and Data Science

Citation

Yue, Daniel, Paul Hamilton, and Iavor Bojinov. "Nailing Prediction: Experimental Evidence on the Value of Tools in Predictive Model Development." Harvard Business School Working Paper, No. 23-029, December 2022.
  • SSRN
  • Read Now

About The Author

Iavor I. Bojinov

Technology and Operations Management
→More Publications

More from the Authors

    • November 2022 (Revised January 2023)
    • Faculty Research

    Hugging Face: Serving AI on a Platform

    By: Shane Greenstein, Daniel Yue, Kerry Herman and Sarah Gulick
    • October 2022
    • Faculty Research

    On Ramp to Crypto

    By: Iavor Bojinov, Michael Parzen and Paul Hamilton
    • September 16, 2022
    • Science

    A Causal Test of the Strength of Weak Ties

    By: Karthik Rajkumar, Guillaume Saint-Jacques, Iavor I. Bojinov, Erik Brynjolfsson and Sinan Aral
More from the Authors
  • Hugging Face: Serving AI on a Platform By: Shane Greenstein, Daniel Yue, Kerry Herman and Sarah Gulick
  • On Ramp to Crypto By: Iavor Bojinov, Michael Parzen and Paul Hamilton
  • A Causal Test of the Strength of Weak Ties By: Karthik Rajkumar, Guillaume Saint-Jacques, Iavor I. Bojinov, Erik Brynjolfsson and Sinan Aral
ǁ
Campus Map
Harvard Business School
Soldiers Field
Boston, MA 02163
→Map & Directions
→More Contact Information
  • Make a Gift
  • Site Map
  • Jobs
  • Harvard University
  • Trademarks
  • Policies
  • Accessibility
  • Digital Accessibility
Copyright © President & Fellows of Harvard College