Making Sense of Recommendations

Michael  Yeomans; Anuj  Shah; Sendhil  Mullainathan; Jon  Kleinberg

Article | Journal of Behavioral Decision Making | October 2019

Making Sense of Recommendations

by Michael Yeomans, Anuj Shah, Sendhil Mullainathan and Jon Kleinberg

Abstract

Computer algorithms are increasingly being used to predict people's preferences and make recommendations. Although people frequently encounter these algorithms because they are cheap to scale, we do not know how they compare to human judgment. Here, we compare computer recommender systems to human recommenders in a domain that affords humans many advantages: predicting which jokes people will find funny. We find that recommender systems outperform humans, whether strangers, friends, or family. Yet people are averse to relying on these recommender systems. This aversion partly stems from the fact that people believe the human recommendation process is easier to understand. It is not enough for recommender systems to be accurate, they must also be understood.

Keywords: recommender systems; artificial intelligence; interpretability; Information Technology; Forecasting and Prediction; Decision Making; Attitudes;

Format: Print

Find at Harvard Purchase

Citation:

Yeomans, Michael, Anuj Shah, Sendhil Mullainathan, and Jon Kleinberg. "Making Sense of Recommendations." Journal of Behavioral Decision Making 32, no. 4 (October 2019): 403–414.

We examine “conversational receptiveness”—the use of language to communicate one’s willingness to thoughtfully engage with opposing views. We develop an interpretable machine-learning algorithm to identify the linguistic profile of receptiveness (Studies 1A-B). We then show that in contentious policy discussions, government executives who were rated as more receptive—according to our algorithm and their partners but not their own self-evaluations—were considered better teammates, advisors, and workplace representatives (Study 2). Furthermore, using field data from a setting where conflict management is endemic to productivity, we show that conversational receptiveness at the beginning of a conversation forestalls conflict escalation at the end. Specifically, Wikipedia editors who write more receptive posts are less prone to receiving personal attacks from disagreeing editors (Study 3). We develop a “receptiveness recipe” intervention based on our algorithm. We find that writers who follow the recipe are seen as more desirable partners for future collaboration and their messages are seen as more persuasive (Study 4). Overall, we find that conversational receptiveness is reliably measurable, has meaningful relational consequences, and can be substantially improved using our intervention.

Article | Proceedings of the National Academy of Sciences of the United States of America | June 30, 2020

Scaling Up Behavioral Science Interventions in Online Education

Online education is rapidly expanding in response to rising demand for higher and continuing education, but many online students struggle to achieve their educational goals. Several behavioral science interventions have shown promise in raising student persistence and completion rates in a handful of courses, but evidence of their effectiveness across diverse educational contexts is limited. In this study, we test a set of established interventions over two-and-a-half years, with ¼ million students, from nearly every country, across 247 online courses offered by Harvard, MIT, and Stanford. We hypothesized that the interventions would produce medium-to-large effects as in prior studies, but this is not supported by our results. Instead, using an iterative scientific process of cyclically pre-registering new hypotheses in between waves of data collection, we identify individual, contextual, and temporal conditions under which the interventions benefit students. Self-regulation interventions raised student engagement in the first few weeks but not final completion rates. Value-relevance interventions raised completion rates in developing countries to close the global achievement gap, but only in courses with a global gap. We find minimal evidence that state-of-the-art machine learning methods can forecast the occurrence of a global gap or learn effective individualized intervention policies. Our findings demonstrate how scaling behavioral science interventions across various online learning contexts can reduce their average effectiveness by an order of magnitude and the promise of iterative scientific investigations to uncover what works where for whom.

Keywords: online learning; behavioral interventions; Scale; Education; Online Technology; Performance Improvement;

Citation:

Kizilcec, Rene F., Justin Reich, Michael Yeomans, Christoph Dann, Emma Brunskill, Glenn Lopez, Selen Turkay, Joseph J. Williams, and Dustin Tingley. "Scaling Up Behavioral Science Interventions in Online Education." Proceedings of the National Academy of Sciences of the United States of America 117, no. 26 (June 30, 2020). View Details

Cite View Details Find at Harvard Read Now

When entering into a negotiation, individuals have the choice to enact a variety of communication styles. We test the differential impact of being “warm and friendly” versus “tough and firm” in a distributive negotiation, when first offers are held constant and concession patterns are tracked. We train a natural language processing algorithm to precisely quantify the difference between how people enact warm versus tough communication styles. We find that the two styles differ primarily in length and their expressions of politeness (Study 1). Negotiators with a tough communication style achieved better economic outcomes than negotiators with a warm communication style, both in a field experiment (Study 2) and in a laboratory experiment (Study 3). This was driven by the fact that offers delivered in tough language elicited more favorable counteroffers. We further find that the counterparts of warm versus tough negotiators did not report different levels of satisfaction or enjoyment of their interactions (Study 3). Finally, in Study 4 we document that individuals’ lay beliefs are in direct opposition to our findings: participants believe that authors of warmly worded negotiation offers will be better liked and will achieve better economic outcomes.

Faculty & Research

Making Sense of Recommendations

Abstract

Citation:

More from these Authors