The Fuzzy Math of Ratings & Reviews

April 2nd, 2009

social-media-monstersMy wife and I took the kids to see Monsters vs. Aliens recently. Seth Rogen as Bob the Blob was funny, and Hugh Laurie was excellent as Dr. Cockroach. A handful of great one liners, but overall a middling effort in the kid movie genre.

The next day – after an appropriate period of contemplation to absorb the deeper lessons of an animated, 3D tale – I asked my wife what she would score the movie on a 5 point scale. She said 2.

I asked her what she would score it on a 10 point scale. She said 6.

I asked her what she would score it on a 100 point scale. She said 70.

2, 6, 70

So what? Well, the statistically-astute readers of Convince & Convert already figured it out, but my wife’s 3 scores – given at the same time, by the same person, for the same movie – are in reality quite different when normalized on a 100 point scale.

2 out of 5 normalizes to 40% on a 100 point scale.

6 out of 10 normalizes to 60% on a 100 point scale.

70 out of 100 of course normalizes to 70% on a 100 point scale.

Thus, it appears that my wife’s appreciation for the cinematic grandeur that it Monsters vs. Aliens jumps by 75% (difference between 40 and 70) based solely on the width of the rating scale.

A 3 Star Hotel, 1 For Each Cockroach

We confront this issue of scale bias every day. Netflix attempts to recommend movies on a 5 point scale (and they have offered a million dollars for improvements to their matching algorithm). TripAdvisor and most other travel sites ask consumers to rate hotels and attractions on a 5 point scale. The Hottie and the Fatso restaurant review blog and podcast authored by my wife and me also uses a 5 point scale.

Interestingly, PriceLine uses a 10 point scale.

Review aggregators like Metacritic and Rotten Tomatoes use a 100 point scale.

As consumers become more and more prone to critiquing everything around them (Forrester research indicates that 37% of U.S. Internet users fall into the “critic” category on their Social Technographics Ladder), a 75% swing in consumer preference based on rating scale could mean real money for businesses.

Imagine a hotel has 9 ratings on TripAdvisor. Four of the ratings are 4 stars. Five of the ratings are 3 stars. The average rating is 3.44, which would round up to a 3.5 score. However, if a 10th rating is added at 1 star, it would bring the average down to 3.20, rounded down to an overall score of 3.

You might give an especially bad hotel a 1 on a 5 point scale. But you would probably not give the same hotel a 2 on a 10 point scale (which is the same, 20% rating). You would probably also not give the hotel a 20 on a 100 point scale. Why? Because 1 out of 5 seems normal. Two out of 10 and 20 out of 100 seem punitive and unnecessary.

What’s the big deal, you say? I researched hotels in Copenhagen recently and the first hotel rated 3.5 on TripAdvisor ranks 16th out of 130 total hotels. The first hotel rated 3 ranks 41st. A potentially significant difference in visibility and bookings.

To me, the 5 point scale is too restrictive and allows for little nuance. And the 100 point scale is too big. Scores almost never fall below 50, because most people innately think of education scoring and its 90 = A, 80 = B scheme when using 100 point scales. The 100 point wine criticism system popularized by Robert Parker is victimized by this scenario. I’ve watched nearly every episode of Gary Vaynerchuk’s WineLibaryTV and even “bad” wines never score below 70 points. In essence, the 100 point scale is really a 30 or 40 point scale from 60/70 to 100. Not optimal.

I prefer the 10 point scale. It provides enough room to make important distinctions, but not so much room it throws off the accuracy of the scale.

Maybe I need to change my restaurant reviews to a 10 point system…

(photo by Jelene)

Want more great content like this?

A weekly dose of the trends and insights you need to keep you ON top, from Jay Baer at Convince & Convert. In each week’s email, Jay will recap what happened in digital, what trends are important for marketers to watch, plus some fun surprises that you’ll just have to sign up to see!

Subscribe to our Newsletter

Or are you looking to subscribe to one of our podcasts