When Do Experts Listen to Other Experts? The Role of Negative Information in Expert Evaluations for Novel Projects

Jacqueline N. Lane; Misha  Teplitskiy; Gary  Gray; Hardeep  Ranu; Michael  Menietti; Eva C. Guinan; Karim R. Lakhani

Working Paper | HBS Working Paper Series | 2020

When Do Experts Listen to Other Experts? The Role of Negative Information in Expert Evaluations for Novel Projects

by Jacqueline N. Lane, Misha Teplitskiy, Gary Gray, Hardeep Ranu, Michael Menietti, Eva C. Guinan and Karim R. Lakhani

Abstract

The evaluation of novel projects lies at the heart of scientific and technological innovation, and yet literature suggests that this process is subject to inconsistency and potential biases. This paper investigates the role of information sharing among experts as the driver of evaluation decisions. We designed and executed two field experiments in two separate grant funding opportunities at a leading research university to explore evaluators’ receptivity to assessments from other evaluators. Collectively, our experiments mobilized 369 evaluators from seven universities to evaluate 97 projects resulting in 760 proposal-evaluation pairs and over $300,000 in awards. We exogenously varied two key aspects of information sharing: 1) the intellectual distance between each focal evaluator and the other evaluators and 2) the relative valence (positive and negative) of others’ scores, to determine how these treatments affect the focal evaluator’s propensity to change the initial score. Although the intellectual similarity treatment did not yield a measurable effect, we found causal evidence of negativity bias, where evaluators are more likely to lower their scores after seeing critical scores than raise them after seeing better scores. Qualitative coding and topic modeling of the evaluators’ justifications for score changes reveal that exposures to low scores prompted greater attention to uncovering weaknesses, whereas exposures to neutral or high scores were associated with strengths, along with greater emphasis on non-evaluation criteria, such as confidence in one’s judgment. Overall, information sharing among expert evaluators can lead to more conservative allocation decisions that favors protecting against failure than maximizing success.

Keywords: Project evaluation; innovation; knowledge frontier; negativity bias; Projects; Innovation and Invention; Information; Diversity; Judgments;

Language: English

Format: Print

46 pages

SSRN Read Now

Citation:

Lane, Jacqueline N., Misha Teplitskiy, Gary Gray, Hardeep Ranu, Michael Menietti, Eva C. Guinan, and Karim R. Lakhani. "When Do Experts Listen to Other Experts? The Role of Negative Information in Expert Evaluations for Novel Projects." Harvard Business School Working Paper, No. 21-007, July 2020.

In summer 2020, Stephane Bancel, CEO of biotech firm Moderna, faces several challenges as his company races to develop a vaccine for COVID-19. The case explores how a company builds a digital organization, and leverages artificial intelligence and other digital resources to speed its operations, manage its processes and ensure quality across research, testing and manufacturing. Built from the ground up as such a digital organization, Moderna was able to respond to the challenge of developing a vaccine as soon as the gene sequence for the virus was posted to the Web on January 11, 2020. As the vaccine enters Phase III clinical trials, Bancel considers several issues: How should Bancel and his team balance the demands of developing a vaccine for a virus creating a global pandemic alongside the other important vaccines and therapies in Moderna’s pipeline? How should Moderna communicate its goals and vision to investors in this unprecedented time? Should Moderna be concerned it will be pegged as “a COVID-19 company?”

True North was a private equity fund that specialized in the growth and buyout of mid-market, India-centric companies. The leadership team initially believed that technology was not core to traditional businesses and steered clear of new age technology-oriented businesses. Then, in 2007, True North invested in Meru Cabs, a ride-sharing service with a growing fleet of cabs that had to change its business model with the launch of tech-based platforms like Uber. This experience led True North to significantly alter its stance on the importance of technology-powered business models. The team learned that all businesses needed to effectively become technology businesses to remain relevant. True North initiated the “Analytics, Algorithms and Artificial Intelligence” (3A) project. A system, Kelp, was designed and developed internally to facilitate the transformation of True North into an AI-first firm based on the introduction of data-driven decision making and productivity-enhancing digital tools. In 2020, as different elements of Kelp were launched, the leadership team met to discuss the future of this technology.

Faculty & Research

When Do Experts Listen to Other Experts? The Role of Negative Information in Expert Evaluations for Novel Projects

Abstract

Citation:

About the Author

More from the Author