Presenting TrueRatr: A Ratings App That Detects Sarcasm

Earlier this year, we shared a post about our third collaboration with Cornell Tech,  which resulted in  a semester long challenge for a team of six students to design and build a product that automates sarcasm detection. This was no easy task. Sarcasm is based on tone and context and as a result can be quite difficult for computers to decipher, as opposed to straightforward rhetoric.

This group  – consisting of engineers, designers and MBA students – worked with Bloomberg R&D, Global Data and User Experience (UX) teams to develop an open source app called TrueRatr, which detects sarcasm for consumer application reviews, like those available on Apple’s App Store.

Apps generate the most number of reviews per product and online reviews drive nearly two-thirds of consumer purchase decisions. So the ability to weed out sarcastic reviews could have a significant impact on future consumer buying/downloading behavior.

As a big believer in data-driven approaches to problem solving and innovation, and a supporter of entrepreneurship in future generations, we were particularly excited to see how the students tackled the issue.

Here’s how the app works:

The students focused on why consumer reviews can be misleading based on language sentiment. Most five star consumer review platforms – like the Apple app store or Amazon  – are  lacking transparency and credibility for authenticity because they accept  anonymous contributions. And, research shows star ratings are not always reflective of what is written in the review, especially for mobile apps.

Initially, due to the lack of data on sarcasm, the team endeavored to start “The Open Sarcasm Project.” This was an attempt to collect sarcastic reviews from users, but unfortunately this method of data collection proved to be too lengthy given this was a three month project. (It would potentially take a year to collect data this way.) As an alternative, data was crowdsourced through Amazon Mechanical Turk.

The team manually sifted through thousands of reviews and using machine learning techniques this data helped to create algorithms that enabled reviews to be defined as either sarcastic or credible.  Taking the next step, using natural language processing and machine learning tools, an algorithm was developed to then prioritize the sentiment of linguistics.

The result:  TrueRatr is able to provide a star rating that removes sarcastic reviews.

Let’s look at some examples:

  • Sifting through the negative: If you search for the video game Grand Theft Auto on the Apple store, you’ll see it has a 4.5/5.0 star rating. If you do the same search on TrueRatr, you’ll see it has a 3.9/5.0 star rating, after analyzing 625 reviews and finding 214 of them contained sarcastic comments.
  • Accentuating the positive: Sarcasm can also have a favorable effect on star ratings. Take Uber for example. It has a 3.5/5.0 star rating in iTunes. On TrueRatr, it has a 3.99/5.0, with 192 of the 609 reviews analyzed being identified as sarcastic and eliminated from the rating.

The TrueRatr for app reviews can be used by anyone, and developers can access the code for the algorithm on GitHub, which could be applied to projects beyond reviews. For more details on TrueRatr and how the Cornell Tech team developed their concept, you can view their slide presentation delivered at the end of their project.

As for any potential use of the technology specifically by Bloomberg – we’ll keep you posted.

Special thanks to the team for their work this challenge. The work was inspired by Christopher Hong – the Cornell Tech student mentor at Bloomberg – and based on his 2014 master’s thesis, which investigated sarcasm detection

Team members included:
Karan Bir, Ming Chen, Hesed Kim, Shreyas Kulkarni, Brendan Ritter, Mengjue Wang

Cornell-Tech-TrueRatr-pic-5

Left to right: Christopher Hong, Rakhee Kher, Mengjue Wang, Ming Chen, Karan Bir, Shreyas Kulkarni, Hesed Kim, Brendan Ritter, Greg Tobkin, Clario Menezes