Skip to Main Content
HBS Home
  • About
  • Academic Programs
  • Alumni
  • Faculty & Research
  • Baker Library
  • Giving
  • Harvard Business Review
  • Initiatives
  • News
  • Recruit
  • Map / Directions
Faculty & Research
  • Faculty
  • Research
  • Featured Topics
  • Academic Units
  • …→
  • Harvard Business School→
  • Faculty & Research→
Publications
Publications
  • 2023
  • Working Paper
  • HBS Working Paper Series

Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality

By: Fabrizio Dell'Acqua, Edward McFowland III, Ethan Mollick, Hila Lifshitz-Assaf, Katherine C. Kellogg, Saran Rajendran, Lisa Krayer, François Candelon and Karim R. Lakhani
  • Format:Print
  • | Language:English
  • | Pages:58
ShareBar

Abstract

The public release of Large Language Models (LLMs) has sparked tremendous interest in how humans will use Artificial Intelligence (AI) to accomplish a variety of tasks. In our study conducted with Boston Consulting Group, a global management consulting firm, we examine the performance implications of AI on realistic, complex, and knowledge-intensive tasks. The pre-registered experiment involved 758 consultants comprising about 7% of the individual contributor-level employees at the company. After establishing a performance baseline on a similar task, subjects were randomly assigned to one of three conditions: no AI access, GPT-4 AI access, or GPT-4 AI access with a prompt engineering overview. We suggest that the capabilities of AI create a “jagged technological frontier” where some tasks are easily done by AI, while others, though seemingly similar in difficulty level, are outside the current capability of AI. For each one of a set of 18 realistic consulting tasks within the frontier of AI capabilities, consultants using AI were significantly more productive (they completed 12.2% more tasks on average, and completed tasks 25.1% more quickly), and produced significantly higher quality results (more than 40% higher quality compared to a control group). Consultants across the skills distribution benefited significantly from having AI augmentation, with those below the average performance threshold increasing by 43% and those above increasing by 17% compared to their own scores. For a task selected to be outside the frontier, however, consultants using AI were 19 percentage points less likely to produce correct solutions compared to those without AI. Further, our analysis shows the emergence of two distinctive patterns of successful AI use by humans along a spectrum of human-AI integration. One set of consultants acted as “Centaurs,” like the mythical half-horse/half-human creature, dividing and delegating their solution-creation activities to the AI or to themselves. Another set of consultants acted more like “Cyborgs,” completely integrating their task flow with the AI and continually interacting with the technology.

Keywords

Large Language Model; AI and Machine Learning; Performance Efficiency; Performance Improvement

Citation

Dell'Acqua, Fabrizio, Edward McFowland III, Ethan Mollick, Hila Lifshitz-Assaf, Katherine C. Kellogg, Saran Rajendran, Lisa Krayer, François Candelon, and Karim R. Lakhani. "Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality." Harvard Business School Working Paper, No. 24-013, September 2023.
  • SSRN
  • Read Now

About The Authors

Edward McFowland III

Technology and Operations Management
→More Publications

Karim R. Lakhani

Technology and Operations Management
→More Publications

More from the Authors

    • September 2023
    • Economic Record

    Consuming Contests: The Effect of Outcome Uncertainty on Spectator Attendance in the Australian Football League

    By: Patrick Ferguson and Karim R. Lakhani
    • 2023
    • Faculty Research

    The Crowdless Future? How Generative AI Is Shaping the Future of Human Crowdsourcing

    By: Léonard Boussioux, Jacqueline N. Lane, Miaomiao Zhang, Vladimir Jacimovic and Karim R. Lakhani
    • July 2023
    • Management Science

    So, Who Likes You? Evidence from a Randomized Field Experiment

    By: Ravi Bapna, Edward McFowland III, Probal Mojumder, Jui Ramaprasad and Akhmed Umyarov
More from the Authors
  • Consuming Contests: The Effect of Outcome Uncertainty on Spectator Attendance in the Australian Football League By: Patrick Ferguson and Karim R. Lakhani
  • The Crowdless Future? How Generative AI Is Shaping the Future of Human Crowdsourcing By: Léonard Boussioux, Jacqueline N. Lane, Miaomiao Zhang, Vladimir Jacimovic and Karim R. Lakhani
  • So, Who Likes You? Evidence from a Randomized Field Experiment By: Ravi Bapna, Edward McFowland III, Probal Mojumder, Jui Ramaprasad and Akhmed Umyarov
ǁ
Campus Map
Harvard Business School
Soldiers Field
Boston, MA 02163
→Map & Directions
→More Contact Information
  • Make a Gift
  • Site Map
  • Jobs
  • Harvard University
  • Trademarks
  • Policies
  • Accessibility
  • Digital Accessibility
Copyright © President & Fellows of Harvard College