Working Paper | HBS Working Paper Series | 2012

Optimal Aggregation of Consumer Ratings: An Application to Yelp.com

by Weijia Dai, Ginger Jin, Jungmin Lee and Michael Luca

Abstract

Consumer review websites such as Yelp.com leverage the wisdom of the crowd, with each product being reviewed many times (some with more than 1,000 reviews). Because of this, the way in which information is aggregated is a central decision faced by consumer review websites. Given a set of reviews, what is the optimal way to construct an average rating? We offer a structural approach to answering this question, allowing for (1) reviewers to vary in stringency (some reviewers tend to leave worse reviews on average) and accuracy (some reviewers are more erratic than others), (2) reviewers to be influenced by existing reviews, and (3) product quality to change over time. We apply this approach to reviews from Yelp.com to derive optimal ratings for each restaurant (in contrast with the arithmetic average displayed by Yelp). Because we have the history of reviews for each restaurant and many reviews left by each reviewer, we are able to identify these factors using variation in ratings within and across reviewers and restaurants. Using our estimated parameters, we construct optimal ratings for all restaurants on Yelp and compare them to the arithmetic averages displayed by Yelp. As of the end of our sample, a conservative finding is that roughly 25%–27% of restaurants are more than 0.15 stars away from the optimal rating, and 8%–10% of restaurants are more than 0.25 stars away from the optimal rating. This suggests that large gains could be made by implementing optimal ratings. Much of the gains come from our method responding more quickly to changes in a restaurant's quality. Our algorithm can be flexibly applied to many different review settings.

Keywords: crowdsourcing; social network; e-commerce; Yelp; Information; Demand and Consumers; Competition; Internet; Reputation; Social and Collaborative Networks; Retail Industry; Food and Beverage Industry;

Citation:

Dai, Weijia, Ginger Jin, Jungmin Lee, and Michael Luca. "Optimal Aggregation of Consumer Ratings: An Application to Yelp.com." Harvard Business School Working Paper, No. 13-042, November 2012. (NBER Working Paper Series, No. 18567, December 2012.)