Are Experts Blinded by Feasibility?: Experimental Evidence from a NASA Robotics Challenge

Resource allocation decisions play a dominant role in shaping a firm’s technological trajectory and competitive advantage. Recent work indicates that innovative firms and scientific institutions tend to exhibit an anti-novelty bias when evaluating new projects and ideas. In this paper, we focus on shedding light into this observed pattern by examining how evaluator expertise in the problem’s focal domain shapes the relationship between novelty and feasibility in evaluations of quality for technical solutions. To estimate relationships, we partnered with NASA and Freelancer.com, an online labor marketplace, to design an evaluation challenge, where we recruited 374 evaluators from inside and outside the technical domain to rate 101 solutions drawn from nine robotics challenges. This resulted in 3,869 evaluator-solution pairs, in which evaluators were randomly assigned to solutions to facilitate experimental comparisons. Our experimental findings, complemented with text analysis of the evaluators’ comments, indicate that domain experts exhibit a feasibility preference, focusing first on the feasibility of a solution as the primary indicator of its quality, while discounting riskier but more novel solutions. This results in a tradeoff in which highly feasible but less novel solutions are judged as being higher in quality, shedding light into why experts prefer more incremental ideas over more radical but untested ideas.