Skip to Main Content
HBS Home
  • About
  • Academic Programs
  • Alumni
  • Faculty & Research
  • Baker Library
  • Giving
  • Harvard Business Review
  • Initiatives
  • News
  • Recruit
  • Map / Directions
Faculty & Research
  • Faculty
  • Research
  • Featured Topics
  • Academic Units
  • …→
  • Harvard Business School→
  • Faculty & Research→
Publications
Publications
  • 2023
  • Working Paper
  • HBS Working Paper Series

Nailing Prediction: Experimental Evidence on the Value of Tools in Predictive Model Development

By: Daniel Yue, Paul Hamilton and Iavor Bojinov
  • Language:English
  • | Pages:75
ShareBar

Abstract

Predictive model development is understudied despite its centrality in modern artificial intelligence and machine learning business applications. Although prior discussions highlight advances in methods (along the dimensions of data, computing power, and algorithms) as the primary driver of model quality, the tools that implement those methods have been neglected. In a field experiment leveraging a predictive data science contest, we study the impact of tools by restricting access to software libraries for machine learning models. By only allowing access to these libraries in our control group, we find that teams with unrestricted access perform 30% better in log-loss error — a statistically and economically significant amount, equivalent to a 10-fold increase in the training data set size. We further find that teams with high general data-science skills are less affected by the intervention. In contrast, teams with high tool-specific skills significantly benefit from access to modeling libraries. Our findings are consistent with a mechanism we call ‘Tools-as-Skill,’ where tools automate and abstract some general data science skills but, in doing so, create the need for new tool-specific skills.

Keywords

Analytics and Data Science

Citation

Yue, Daniel, Paul Hamilton, and Iavor Bojinov. "Nailing Prediction: Experimental Evidence on the Value of Tools in Predictive Model Development." Harvard Business School Working Paper, No. 23-029, December 2022. (Revised April 2023.)
  • SSRN
  • Read Now

About The Author

Iavor I. Bojinov

Technology and Operations Management
→More Publications

More from the Authors

    • August 2023 (Revised August 2023)
    • Faculty Research

    Sparking Innovation in the U.S. Air Force

    By: Michael Parzen, Alexander Farrow, Paul Hamilton and Jessie Li
    • July 2023
    • Management Science

    Design and Analysis of Switchback Experiments

    By: Iavor I Bojinov, David Simchi-Levi and Jinglong Zhao
    • 2023
    • Faculty Research

    Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Online Experimentation

    By: Dae Woong Ham, Michael Lindon, Martin Tingley and Iavor Bojinov
More from the Authors
  • Sparking Innovation in the U.S. Air Force By: Michael Parzen, Alexander Farrow, Paul Hamilton and Jessie Li
  • Design and Analysis of Switchback Experiments By: Iavor I Bojinov, David Simchi-Levi and Jinglong Zhao
  • Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Online Experimentation By: Dae Woong Ham, Michael Lindon, Martin Tingley and Iavor Bojinov
ǁ
Campus Map
Harvard Business School
Soldiers Field
Boston, MA 02163
→Map & Directions
→More Contact Information
  • Make a Gift
  • Site Map
  • Jobs
  • Harvard University
  • Trademarks
  • Policies
  • Accessibility
  • Digital Accessibility
Copyright © President & Fellows of Harvard College