You are previewing The Problem With Online Ratings.
O'Reilly logo
The Problem With Online Ratings

Book Description

In the digital age, we are inundated with other people’s opinions. We browse books on Amazon with awareness of how other customers liked (or disliked) a particular tome. On Expedia, we compare hotels based on user ratings. On YouTube, we can check out a video’s thumbs-up/thumbs-down score to help determine if it’s worth our time. For the most part, consumers have faith in online ratings and view them as trustworthy. But, the author argues, this trust may be misplaced. The heart of the problem lies with our herd instincts — natural human impulses characterized by a lack of individual decision making — that cause us to think and act in the same way as other people around us. When it comes to online ratings, our herd instincts combine with our susceptibility to positive “social influence.” When we see that other people have appreciated a certain book, enjoyed a hotel or restaurant or liked a particular doctor, this can cause us to feel the same positive feelings and to provide a similarly high online rating. The author describes an experiment that he and two colleagues conducted on a social news-aggregation website. On the site, users rate news articles and comments by voting them up or down based on how much they enjoyed them. The researchers randomly manipulated the scores of comments with a single up or down vote and then measured the impact of these small manipulations on subsequent scores. The results were striking. The positive manipulations created a positive social influence bias that persisted over five months and that ultimately increased the comments’ final ratings by 25%. Negatively manipulated scores, meanwhile, were offset by a correction effect that neutralized the manipulation: Although viewers of negatively manipulated comments were more likely to vote negative (evidence of negative herding), they were even more likely to positively “correct” what they saw as an undeserved negative score. This social-influence bias snowballs into disproportionately high scores, creating a tendency toward positive ratings bubbles. Positively manipulated scores were 30% more likely than control comments (the comments that the researchers did not manipulate) to reach or exceed a score of 10. A positive vote didn’t just affect the mean of the ratings distribution; it pushed the upper tail of the distribution out as well, meaning a single positive vote at the beginning could propel comments to ratings stardom.