Hello, Cafe --
Back in Nov.-Dec. of 2010, I complained vigorously to Yelp's customer
service dept. about what I see as a case of bad information design ...
and when I felt like our discussion was going nowhere fast, I told
them I'd move my yelping over to an international discussion list for
ID professionals, where I knew I would get much better arguments &
feedback, plus an opportunity to learn whether my negative reaction to
the Yelp review model and website is peculiar to me, or is more
broadly shared.
I've been putting off starting this discussion because I've been
working even more intensely than usual, trying to launch 3 new
websites (2 down ... one still to go! ;-), but decided it was time to
make it a priority when yesterday I heard yet another related report
on the US public radio program, _Marketplace_:
"The power of the online product review"
originally aired: 18 Jan. 2011
<
http://marketplace.publicradio.org/display/web/2011/01/18/pm-the-power-of-the-online-product-review/
>
So, to get us started, here's my Yelp story....
My own troubles with Yelp began back in May 2010 when I set out to
write an online review for the veterinarian my family has been going
to for over 20 years. He had recently performed another outstanding
surgery on my elderly dog (in this case, for a large mouth cancer
which had ruptured), and after I finished telling him in detail how
pleased we all were with the outcome, he asked me if I would consider
writing an online review.
Ordinarily, I wouldn't do this (I have no time for, or interest in,
writing product reviews), but this was an extraordinary occasion ...
made even more so when I actually looked up his business online.
To quote from the Yelp review I later wrote for his veterinary practice,
"I have a hard time recognizing the skilled and compassionate
vet I've known for 2 decades in some of the negative reviews
here and on other Web pages."
Not only were some of the reviews quite nasty little rants, but at
Yelp, the negative reviews were given more prominence and weight. I
really had to poke around to find the 5-star reviews -- almost all of
which had been "filtered" (meaning, they are removed to a separate
page, which you access via a link in small print at the bottom of the
business's main Yelp page, after which you have to decipher a bunch of
skewed type before the page with filtered reviews will display).
If/when you make it this far, a notice in small gray type (easy to
miss when you're scanning the Web page for the content you're after)
informs you that
"Note: The reviews below are not factored into the business's
overall star rating."
Well, in my vet's case, this is kind of a big deal. There are (as of
today) 6 reviews of his veterinary practice in this "filtered"
category. Five of these (including my review) award him 5 stars; the
other filtered review gives him 1 star. But none of these 6 reviews
contribute to the rating his business gets at the top of the main
page. What we see there is an overall 3-star rating (with an ad 2
inches below this for another area vet with a 4.5-star Yelp rating).
When I first wrote my Yelp review back in May 2010, I was concerned
that my 5-star review might be filtered, as had so many others, so I
read everything I could find about how to write a proper Yelp review,
so that there would be no question about its legitimacy. And
apparently there wasn't ... at first.
My review appeared right there at the top of his main page as soon as
I posted it, so I thought I'd been successful, and moved on to other
projects. What I didn't know was that Yelp's treatment of reviews is
dynamic. Yelp actually alters the status of reviews *after* they're
posted, without warning or notification.
I only discovered this when I returned to update my vet's review 6
months later, the day after my dog was euthanized.
At the time, emotions were pretty raw, and I was not pleased to
discover that my review had been filtered and judged a "fake" by the
omnipotent Yelp algorithm which reviews the reviews.
I hoped that updating my review would change my "fake" rating, but it
didn't make any difference.
As a consequence, I wrote to Yelp to complain, and even this required
a certain amount of ingenuity in order to bypass the (designed-in ?)
limitations of their form for feedback. Among other things, I had to
divide my comment into 3 parts, each of which was transmitted as its
own response. Here's what I wrote in the 3 separate e-mail/forms:
1.
(Part 1 of 3)
My comments on improving Yelp all relate to the fact that my
review of a local business has been "filtered" and made
nearly inaccessible for reasons that are unclear to me,
beyond the fact that I am not -- and have no intention of
becoming -- an active Yelper. I was only persuaded to join
Yelp in the first place because I felt that my 5-star
experience and point of view was completely lacking from the
reviews for the [business name] in [business location],
posted to [URL for business's Yelp page]. Since my opinion
still isn't properly represented, I see no reason to waste
further time writing even more Yelp reviews that apparently
don't count.
I understand that there are major issues with people trying
to game the system, and that Yelp has a genuine need to
somehow filter reviews. But if my experience is any
indicator, your filter does NOT do "a pretty good job" of
this, as claimed in your FAQ.
(continued)
2.
(Part 2 of 3)
I am a real, long-time customer of the [business name], and
my review of the business is not a fake. When I wrote my
review, I went to great pains to set up my Yelp account with
an avatar, and a website address, which makes it quite clear
that I am who I say I am. Plus, I was careful to write
original and different reviews for the [business name] on
Yelp vs. Google Maps, etc., so that my Yelp review of
[business name] would not be filtered out because it was
repetitive and/or available elsewhere.
Moreover, I figured I'd been successful in this because my
review originally appeared on the main Yelp page for
[business name] directly after I posted it, and I was never
informed that there was a chance it might be arbitrarily
"filtered" out at some future point. It was only when I
returned today to update my Yelp review of 6 months ago that
I discovered it had been so filtered.
(continued)
3.
(Part 3 of 3)
My actual complaint is not just that Yelp altered the status
of my review without notifying me, but also that Yelp's
ratings data pretends to be representative of all those who
have yelped about a business, when in reality it is selective
& skewed -- but without any warning that this is the case,
and that some points of view have been mechanically censored
using some mysterious algorithm which might very well result
in mistaken exclusions. At the very least, there should be a
decent-sized link to the filtered reviews at the bottom left
of each Yelp page, accompanied by a brief warning that the
above ratings are not compiled from ALL the reviews Yelp has
received, but only from a selected few, with message along
the lines of "click here to read the censored reviews".
As it is, my limited experience with online review sites such
as Yelp merely confirms all the negative criticisms of Yelp
I've encountered elsewhere (e.g.,
http://marketplace.publicradio.org/display/web/2009/04/10/pm_yelp/ ).
This drew a response from Customer Service at Yelp directing me to a
video explanation of their review filter at
http://officialblog.yelp.com/2010/03/yelp-review-filter-explained.html
which I found full of questionable assumptions.
I watched the video, and read all the related Web pages accessible
from the above URL, and still didn't feel that my real issues had been
addressed.
So, I replied with the following:
[...] As far as I'm concerned, Yelp has every right to filter
reviews ... and to even brag about the filter you've devised
for doing so.
But I think it is misleading (bordering on deceptive) not to
properly warn people -- up front, in legible type, preferably
right next to the starred rating -- that
1. Yelp reviews are *mechanically* filtered.
2. Filtering is dynamic, so that the exact same review may be
judged "true" one day, and "fake" the next.
3. A business's starred rating does NOT include any
reviews which have been filtered.
4. A business's starred rating changes NOT just because new
reviews have been posted, but because previously "true"
reviews may at any point be filtered out as "fake", and vice
versa.
Plus, I think this notice should include a clear, running
tally of how many reviews (of the total received) were used
to calculate a business's rating, and how many reviews were
excluded from the ratings process (i.e., filtered).
I would argue that this metadata is necessary because it's
not just the Yelp cognoscenti who are reading (and probably
relying on) Yelp reviews.
Now that the search engines deliver up Yelp ratings to all
sorts of people who (like me) know nothing whatsoever about
Yelp and its unique ratings system, I believe Yelp has a
responsibility to inform casual visitors that our
expectations about how a starred rating is calculated (i.e.,
the average of *all* received reviews) don't apply at Yelp.
Had I known this from the beginning, I wouldn't be
complaining now about Yelp's filtering mechanism having
misjudged my review.
In sum, I don't care how opaque your filtering process is, or
what "trust factors" you use to assess reviews.
What I do care about is that we're not told up front that
reviews are mechanically filtered, with ratings skewed
accordingly.
I'm willing to bet that most people have no idea Yelp filters
reviews -- or even that fake reviews are such a problem.
I certainly didn't.
Nor do I believe that Yelp's filtering mechanism -- or any
filtering mechanism, for that matter -- can guarantee
objectivity when it comes to assessing human motives.
IMO, the best judge of human behavior (including deliberate
deceit) is human intelligence, critically applied.
Critical human judgment is not perfect, of course. But it
works well enough for most of us to be able to maintain a
satisfying social life, and even join with others in the
pursuit of larger civic goals.
Surely if we humans can manage this much, we can also figure
out which online reviews to trust temporarily, until variable
personal experience proves all involved either right or
wrong. [...]
I probably should add here that there are a few things about the Yelp
review model that I do like, such as the fact that Yelp allows
reviewers to post updates (which I wasn't able to do elsewhere --
Google Maps, Bing Local, Kudzu), and also that they allow longer
reviews -- 5000-character limit at Yelp, 4096-character limit at
Google Maps, 4000-character limit at Yahoo Local, 2000-character limit
at Bing Local (which also can't count properly! ;-), and a 250-word
limit at Kudzu.
I should also say that, once I was able to access it, Yelp's customer
service beat the pants off of that provided by Yahoo Local, which was
so bizarre, it bordered on surreal, and I ended up feeling like I was
trapped in some kind of endless computer loop ... although that would
imply some sort of rationality that was completely missing from my
Yahoo encounter! After 6 or so canned replies that made no sense, I
grew so irritated with Yahoo customer service that by the time a
thoughtful human being finally came to my aid, I had already
proclaimed to all & sundry that I was done with all things Yahoo, and
would NOT be posting my review to their site ... ever.
(And, of course, there's no going back on one's public ultimatums, no
matter the cost, or there's no point in issuing them in the first
place! ;-)
In sum, my little odyssey into the online world of product/service
reviews left me questioning the whole genre.
I think there're a lot of problems with the *business* of online
reviews, and what bugs me most about it, is the (sometimes extreme)
partiality masquerading as universal truth.
Perhaps the behavioral economists have some answers.
I'm hoping some of our Café mathematicians can explain the issues,
too. Every teacher knows that our grading systems are always full of
flaws, which are then rendered more or less moot by (predictable ?)
human factors.
... Maybe we just trust to _The Wisdom of Crowds_ (yes, Surowiecki's
book, but others toting the virtues of data mining for decision-making
processes, too) to balance everything out.
But isn't group wisdom really a matter of scale?
When you only have 15 reviews of a business, with almost as many
underlying value systems -- reviewers who give a 1-star rating because
they think the color of the chairs in the waiting room sucks, casually
intermingled with those who award 5 stars only when a vet raises their
dog from the dead -- I'm not sure the mathematics of a simple, 5-star
review model work well at all ... and especially not when websites
like Yelp turn the 5-star rating system over to the machines....
Deborah
_____
Deborah Taylor-Pearce
***@she-philosopher.com
___________________________________________________________________
Use the following address to post a message to all subscribers:
infodesign-***@list.informationdesign.org
To subscribe, unsubscribe or change your options, visit:
http://list.InformationDesign.org/mailman/listinfo/infodesign-cafe
For all Information Design matters:
http://InformationDesign.org
Problems? Write to:
InfoDesign-Cafe-***@list.InformationDesign.org
___________________________________________________________________