IntroductionAre reviews primarily a consumer guide, or should they serve another purpose? Do review scores deter intelligent discussion of videogames? Is the presence or absence of a review score the only difference between a reviewer and a critic? What is the role of the reviewer when the Internet is democratizing published opinion? How should reviews and reviewers evolve in light of the emergence and growth of Flash games, small games, indie games and user-generated games?
These questions and more were on the mind of N'Gai Croal, John Davison and Shawn Elliott last summer when they decided to expand their conversation to a number of noted reviewers, writers, bloggers and journalists for a published email symposium on game reviews. (See below for the full list of participants.) The planned list of topics include Review Scores; Review Policy, Practice and Ethics; Reader Backlash; Reviews in the Age of Social media; Reviews in the Mainstream Media; Casual, Indie, and User-Generated Games; Reviews vs. Criticism; and Evolving the Review. Round 1's topic: Review Scores.
Participants
Leigh Alexander, Gamasutra/Sexy Videogameland/Variety Harry Allen, Media AssassinRobert Ashley, freelancerTom Chick, freelancerN'Gai Croal, Level Up/Newsweek John Davison, What They Play Shawn Elliott, 2K BostonJeff Gerstmann, Giant Bomb Kieron Gillen, Rock, Paper, ShotgunDan Hsu, Sore Thumbs BlogFrancesca Reyes, Official Xbox Magazine
Stephen Totilo, MTV NewsReview Scores
Shawn Elliott, 2K Boston: How much is on our minds before we begin playing any given game for review purposes? Will we imagine a range of probable scores that a heavily marketed, highly budgeted, and hugely anticipated game will get? What when the game is branded “budget” or is the work of a lesser-known, less-storied studio? If so, how closely have actual scores correlated with our assumptions?
Kieron Gillen, Rock, Paper, Shotgun: As others have said before—but Troy Goodfellow put most snappily, so I'm stealing his phrasing—the games press has a presentist/futurist bias. The vast majority of press coverage is for games that either aren't available, or are only just available. Even if we haven't seen or played the game personally, our peers will have. And we'll have seen comments threads full of people saying what *they think* of the edited information of the game we (and their PR) have presented. And with all that, when you throw a score out you know it's going to be read with those expectations in mind. When Eurogamer's Metal Gear Solid 4 review gave it an 8/10 there were 2000-post threads and actual death-threats. And Oli [Welsh], when he wrote that review, knew exactly what response he could expect. Games without the hype have lower expectations. I remember the attitude being crystallized by a comment I saw ages ago on Kotaku which stuck with me, when they linked to a B-game someone had 9/10ed: "It can't be any good, as I haven't heard of it". It's an ugly, but common, tautology.
You can't avoid knowing what the score is on that point, without becoming a true hermit. In terms of coloring your actual expectations of the game per se... well, unless someone's actually paying me to research a feature, I ignore 95% of previews. So when reviews come up, I try to review what's there rather than the hype... but that's going onto a whole different question.
Quick thought regarding the indie/AAA dichotomy, though: I often think that AAA-popular-sequels tend to start with 9/10 and lose marks, while games with less expectations start with 5/10 and have to gain them. And... oh, I'll shut up. More on this later, I suspect.
***
Leigh Alexander, Gamasutra/Sexy Videogameland/Variety: So, as far as preconceptions go, I just thought it worth noting that a game's marketing machine, whether through its fierceness or its clumsiness, would very much for like for us to have a preconception going into a review.
Unfortunately for them, they can't necessarily pick what impression they create. I like to think we react to the fashion in which we're being messaged, rather than devouring piecemeal the messaging itself. Or, most of us do.
So, I agree with Kieron that the right answer is "no preconception"—i.e, the reviewing process doesn't begin until you start playing the full version of the game, period. But sometimes I wonder whether background factors should be considered as context for a review. For example, for months a hyperbolic individual promises that his game will revolutionize ludology. Are we allowed (or, conversely, obligated?) to consider his lofty goals when evaluating the end result? If a company creates an "identity" for a game ahead of time, shouldn't that exemplify what the game is aiming to be, and shouldn't we try and consider whether or not it achieves it?
There's a line, I think, between making a prejudgment, and bringing with you a context within which to make an evaluation. Games are an industry and a culture, not a fragmented, compartmentalized list of disparate products, and rather than pretend we have no early opinions, I wonder if it's not beneficial to be prepared to bring that context—which also applies, perhaps to being aware of budgets, of team sizes, of other challenges?
***
Shawn Elliott, 2K Boston: Because I believe that self-enhancing, self-serving, egocentric biases are normal, and that people are prone to see themselves as being immune to the influences that move everybody else, I'll happily admit—along with Kieron—that I have preconceptions before playing. I'm human.
I'd argue that our preconceptions are active when we decide which games we want to review. That's not to suggest that, when given the choice, all critics go straight for the gravy (I've often volunteered to review games that I imagined would be interesting but not the best available). But what, if not a preconception of some sort, drives these decisions?
In addition, I believe that my assumptions are active as I play. For instance, I'm less likely to immediately doubt the wisdom of a given design choice in a Valve game than I am with the work of second-rate studios. An analogy: Say you're competing against someone with sorry win-loss stats in a strategy game. His opening moves seem odd, so you assume he's stupid. When his record is intimidating, you take the time to study his seemingly odd tactics until you're certain you're not missing something. In my mind, "the right answer" isn't a realistic answer.
Leigh, I have a problem with holding a loud developer to his hyperbolic promises (and it has nothing to do with the dozens of programmers, designers, producers, artists, and animators hanging their heads behind him): intentional fallacy. I'm interested in the degree to which game maker's games match their ambitions, but I wouldn't want to evaluate them on this basis. What New Critics wrote of poems seems sensible for games: "It is detached from the author at birth and goes about the world beyond his power to intend about it or control it. The poem belongs to the public."
Should we consider budgets and staff sizes? Certainly not when the critic's intent is strictly to inform consumer shopping sprees.
***
N’Gai Croal, Level Up/Newsweek: I’ve never liked assigning scores as part of any critical assessment, and the times I’ve had to do so in the past, it’s always been under duress. I started out as a journalist by writing movie reviews for my college paper, and none of the critics after whom I tried to pattern myself—Pauline Kael, J. Hoberman, Stanley Kauffmann, John Simon, Andrew Sarris, Armond White—used stars or points or thumbs. They didn’t provide you with any shortcuts or shorthand. You had to read what they wrote in its entirety in order to figure out what they thought. I said to myself, when I grow up, that’s the kind of critic that I want to be. So because I’m not obligated to dole out review scores in print or online, I only have two things on my mind when I start playing a game that I know I’m going to write about.
First, am I going to enjoy this game? In that sense, it’s not dissimilar from when I take in a movie. Or a TV show. Or a play. Or a book. Even when it’s a shared experience, playing a game is intensely personal, and no matter the developer’s pedigree, no matter the budget, I start each new title the same way: on the precipice between hope and fear. I hope that it will be good or great. I fear that it will be mediocre or worse. And as I give myself over to that series of firsts—the first image, the first sound, those first bits of gameplay, that first accomplishment—any and all external influences evaporate, leaving me only the thrum of my internal gauge, the one that tells me just how much I’m enjoying myself. I trust that gauge implicitly, and while external factors might influence precisely how I /articulate/ my opinion, I don’t believe it goes much beyond that.
Second, how much of this game am I going to be able to complete before my deadline? That’s very different from how I approach plays, television, theater or literature—I wouldn’t dream of critically assessing a piece of work from those media without having completed it. Why doesn’t that stop me from doing the same with videogames?
The explanation—or is it an excuse?—that I offer is that I don’t review games. We’ll get into this more in the Reviews vs. Criticism section of our symposium, but the way I see it, a reviewer answers the question, how well does this game work, but a critic answers the question, how does this game work? A reviewer helps consumers decide whether or not they should buy a game; a critic helps players think about a game that they’ve played—in its entirety /or/ in part—and that is the end of the spectrum where I believe my writing lies. (That’s also why, on a game by game basis, I don’t think I need to have completed a game to have some insights about it—but I do think that if I were advising someone on how to spend their money, I’d feel obligated to play most or all of the game.) Scores can serve as a valid form of shorthand for the work of the reviewer, but I’m not convinced that scores have much to offer the work of the critic.
***
Kieron Gillen, Rock, Paper, Shotgun: Leigh, I agree with Shawn. You can mention the hyped intention and mention whether it measures up—but that's not what you're rating. Marketing doesn't necessarily understand their games and what's interesting about it. And occasionally a game is fascinating despite what their creators were trying—Jim Rossignol loving the deeply buggy unpatched release of Boiling Point for its sheer constant surreality comes to mind as an extreme example of that.
N'Gai, it's far too early for me to do my You Don't Need To Complete A Game To Review It piece, I suspect. Methodology of reviews is a question all of itself.
***
Stephen Totilo, MTV News: I wonder why Shawn dragged me into this. I seldom write reviews. I don't put scores on games. My main gig's reporting, a.k.a journalism, a.k.a. the thing most people don't really mean when they want to talk about "games journalism" because the thing they really mean to muddle over and improve upon is what we're talking about here: games-reviewing. I'll give it a go, nonetheless! Scores, who are they for? What do they do?
The question we're answering is whether those who review games pick a number before writing a word. Kieron says the ideal reviewer would not; he and Leigh agree it's hard not to pick a figure already. Shawn's acknowledging the humanity of having preconceived notions but dodging his own question about whether that made him start with a number. But I guess it's hard in some ways to pick a figure at all when it's so unclear what the point of it is.
What does it mean to select -- prematurely or even at the "right" moment—a seven for a game? Or to see a game and, at first sight, have your gut gurgle that it’s a nine?
A review score number may be for the fans, a shopping guide metric that informs a purchase or justifies one already made. It may get used for the dastardly purpose of comparing a game to another—even though it never quite works to pit a 2008 sports game that got an eight against a 1998 role-playing game that got a nine, especially if neither is as good as Tetris. A numerical score might, in isolation, even indicate if a game's any good, but not always.
We're talking about arriving at a number, and, frankly, I don't know how you all do it. A decade ago I worked at a boxing magazine and sat in press row for many fights. Scoring vexed me then. I'd score rounds for my coverage on the "10-point must" system: 10 for the winner of the round, nine for the loser unless he got knocked down or really took a beating, which would dock him to an eight. In that system we see the Gillen-described method of scoring-by-reduction. We also saw the great gaming tradition of grade inflation. Give a judge (or a reporter aping the actions of the official judge) a 10-point scale and all kinds of psychology comes into play.
The other thing I saw at the fights—the thing that really stuck with me—was how hard it was to score any of it. Boxing matches aren't like Rocky fights. It's often hard to see who is winning or which fighter is doing the better work. Sometimes it's all boring or repetitious, but you still must score each three-minute round. Putting numbers on these things—and the official judges had to, in case it went the distance and, god forbid, the paying public needed to know who won—was a murky and unpleasant job. Try it some time. I'd root for the knockout, which would render scores moot and sweep any errors in numerical judgment away. The scorecards didn't matter then. Any scoring biases we had would be secret. The fallacy of putting a number on things would be dodged, and everyone would go home happy. No one would have to know that I gave a 10 to fighter B because I felt bad that he'd gotten beaten up for the three previous rounds or that I gave the wrong guy the first round because I bought into his pre-fight hype.
***
Robert Ashley, freelancer: I took a break from enthusiast press game reviews for a couple of years. What a fucking relief. No more death threats from insane superfans who think my evaluation of their favorite game is some kind of paid-for hit job by a shadowy corporate network. No more forcing myself to play through a 40-hour game in three days. No more tearing my hair out trying to avoid the clichéd language of a form of writing frozen in its awkward adolescence 15 years ago. Free to play whatever I wanted, I fell in love with games all over again. Hard.
Now that I'm back and picking up the occasional review, I simply refuse to engage in the bullshit that used to drive me insane. Review scores have one use: driving traffic from message boards and social networks to your site and giving those people an excuse to argue out their fan beefs in the comments section. I treat them as such.
I have no methodology for choosing a review score. I certainly don't think about it much. Your gut feeling (after either beating the game or the game beating you) is more accurate than whatever you might come up with after careful consideration. This is how the rest of the gaming community arrives at an opinion—and probably why so many people feel that critics are out of touch. When you sit at your computer, running down all the plusses and the minuses—technical issues, story concerns, lovable roughness, annoying roughness—you can end up talking yourself into a score that doesn't really represent your true reaction. You can't explain the magical pixie dust that made the empirically bad game good. You can't explain the soullessness and sterility that made the empirically good game bad. You let your stupid logical brain take the wheel and explain yourself into a lie.
When I say you, I mean me.
Anyway, I say be gutsy and honest with a score, and save your careful thinking for the text.
***
Jeff Gerstmann, Giant Bomb: Well, I won't deny that scores stir up message boards and social networks and such. But to claim that's the only reason they exist is a pretty narrow, jaded view. I think scores are primarily there to serve as shorthand for folks that won't or can't read the full review. They're meant to serve as part of the summary. A deck, a score, and, depending on your publication's review style, some pros and cons or whatever. They aren't rocket science, and were never really meant to be treated as such. The key is to not let the different ways that scores are misused get in the way of what you're trying to accomplish with your reviews. I don't care if the scores I give fit in with the rest of the industry on the review aggregator sites. I don't care if people infer the score to mean that I'm playing favorites because I'm obviously "TEH BIAS" or whatever. I care about the people out there who haven't been following a game from day one, and the people who haven't already pre-ordered the game and are just looking for validation. As soon as you start bending your review systems in order to cater to those extremist segments of the audience, you're getting away from the thing that reviews are designed to accomplish: assist average, everyday people in their purchasing decisions.
I say assist because we've reached a point where one review can't possibly work for every single person that reads it. The audience for video games is too widespread and varied now for reviewers to think that their review is the only one that matters, or that it will be able to directly state if a person should or shouldn't buy a game. This, more than anything, is what should be driving a change in the way games are reviewed, not a bunch of reviewers who are tired of all the weak-ass game review clichés that are still out there. Getting rid of scores because people who write reviews are tired of assigning them and dealing with the fanboy rage that invariably ensues hurts the consumers that actually use reviews for their intended purpose.
But to answer the core questions, I don't really think too much about scores when I'm playing a game. I attempt to go in feeling cautiously optimistic about the game in question, and as I'm playing, I think about text, and things in the game that need to be specifically called out. I start to think about the best way to mention those moments, and the best way to call out its flaws. At some point, all that text swirling around in my head starts to sound like a range of scores, so maybe around halfway through playing a game I start thinking a little more about the score. But it isn't until after the review is written that the score is actually assigned. The score is meant to sum up the text. If I've just written a review full of harsh criticisms, well, then that sounds like a pretty low score. Assigning a score and then attempting to justify it with text puts the cart before the horse.
Assuming a score (or range of scores) before actually playing the final game is pretty dangerous territory. Carefully controlled publisher-run demos usually paint a pretty rosy picture of a game, and games often don't live up to that. Case in point: every time I saw Mercenaries 2 prior to its release, I thought it looked awesome. The missions seemed smart, the co-op was fun, and it felt like a game that would offer a lot of variety. The final product turned out a collection of dopey missions that showcased the game's boneheaded AI, the co-op didn't make much sense, and a lot of the missions were pretty boring. I didn't review Mercs 2, but not letting pre-release exposure to a game color your review with overt disappointment or a sense of smug "I totally called it" satisfaction can get a bit tricky.
So I agree that, ideally, a reviewer should start with no preconceived notions about a game based on budget, hype, promises made by the developer, and so on. But at the end of the day, we're all human, and I'd expect that some form of disappointment over a game that fails to deliver on promises or excitement over a sequel that's turned out better than the last leaks into some of our reviews. The key is in owning up to that and presenting your reviews as informed opinions, rather than hiding behind the old paradigm of rigid objectivity.
***
Shawn Elliott, 2K Boston: I didn't mean to duck the question, Stephen, and I definitely don't start with a specific rating in mind. However, I'm sure that I have imagined ranges of scores that a given game would receive whether I or anyone else was to write the review. That's not to suggest that I once forced the square peg of a game to fit the round hole of my presumptions. I never did. Or I don't think I did. What I'm acknowledging is that, all the same, something was on my mind, both before I began and while I was playing. I think this is the case for every videogame critic. And while that something isn't necessarily decisive, it's nonetheless worth investigating.
I should also add that our predictions regarding meta-ratings and the reviews of other critics are on the mark more often than not. (In these instances, self-fulfilling prophecy isn't an issue.) Some companies are so confident of our ability to make these calls that they're willing to pay us for our input as consultants.
Jeff is correct in that sometimes PR-controlled preview demonstrations are smoke-and-mirrors magic shows. But what about when we're allowed to play near-complete code for prolonged periods? I'm not talking about performance issues—commenting on the framerate of an unfinished game is almost as pointless as it is for an Entertainment Weekly writer to assure her audience that King Kong may or may not appear in place of a green screen. Sometimes design, locked down years prior to a game's preview phase, is apparently dopey. Again, I have to emphasize that holding some assumptions in no way necessitates my maintaining them in the face of final evidence.
You also imply that an aversion to cliché shouldn't drive change in the way that we review games. I won't argue that cliché is the one and only reason to reconsider our habits, however, I count it among the many. The paragraphs on a game's graphics, sound, and so on in previews and reviews produce recognizably generic writing devoid of the discovery and perception that might make them worth reading. They are lazy in that they eliminate both the need to transition thoughts and to interpret a game as the complex product of interconnected components (instead of simply summarizing these parts).
Even worse is when the paragraphs that constitute a template are themselves composed of yet more methods of avoiding actual analysis. I mock the overuse of words such as compelling not because there is anything wrong with the words themselves but rather with the way that they're used to replace real explanation. We know that any guy in the game store can say he likes or doesn't like a game's graphics or story. We recognize that it's our responsibility as paid writers to say something more than "I like it" or "it's good." Replacing "like" and "good" with "compelling" isn't even trying.
***
John Davison, What they Play: If nothing else, review scores serve as the starting point of a discussion for readers. As Jeff says, they serve as a shorthand for those that have no interest in digging deeper than a fundamental thumbs up or thumbs down gauge of quality. I think we can all safely assume this, but back in my time at Ziff we experimented sufficiently that we got absolute, empirical proof.
Jeff Green and I spent a lot of time talking to Computer Gaming World readers, and trawling through our message boards to really try and put together the ultimate reviews section for the audience. We wanted to do something a bit different, but more than anything we wanted to acknowledge what a large group of our readers were telling us. That was, essentially, that "we're older" and "we're smarter" than the average gamer, so "treat us like that." They wanted longer, more considered think pieces about games, and it appeared, anecdotally at least, that review scores were not high on their list of priorities. They wanted, they said, to really understand what the reviewers were trying to convey. They wanted to really dig in.
So we gave them that. We took the scores off, and made the reviews longer. We actually went a step further, and tried to acknowledge the broader critical spectrum, and talk about what caused other reviews to express particularly positive or negative comments. It was our own little expression of idyllic critical idealism. A utopia of reviewing and we dreamt that it would spark enlightened and intelligent debate about specific qualities and opinion.
The reaction was spectacular. The readers really, really fucking HATED it. The most common complaint (I'm paraphrasing, but it was pretty consistent) was "How do I know what you think if you don't give it a score?" That and "you guys are retarded." We figured at first that it was simply a bit of culture shock and that it would wear off, but the negativity increased over time. After three months or so, we had to go back to putting a score out of five on the reviews just to stem the tide of vitriolic hatred.
On a separate note, I was speaking to someone recently who had some connection to Rolling Stone, and he told me that the reviewing process for albums there was that the critics only submit the text, but do not submit a score. The number of stars is assigned by the reviews editor based on the tone of the review. He was drunk at the time, so might have been talking out of his arse though. Does anyone know for sure if this is the case? Even if it's not true, it's certainly an interesting approach—and something I'd like to discuss in this context. If a reviewer is freed from thinking about assigning a score, but knows one will be applied later—would it necessitate a more disciplined approach to how thoughts are expressed? I know it would for me. But are we ready to relinquish that kind of control?
***
Robert Ashley, freelancer: I don't advocate putting an end to scoring. Scores seem to be the one thing that today's online audience can easily form a conversation around, and I think, ideally, a review should be like a conversation between reader and critic. I just think the incredible seriousness surrounding scores (born in no small part, I'm guessing, from the fact that retailers stock their shelves based on review scores, ratcheting up pressure on critics to treat scores like jury sentences) is irritating. Handing out a mediocre score to a mega-hyped game can brand you a heretic (a Crispin Boyer, if you will) when you're just trying to be honest about your reaction.
***
Stephen Totilo, MTV News Multiplayer: Who is actually upset about review scores?
Offended publishers who wanted a 9? Stressed developers whose bonuses depend or ability to get another deal require them to get at least an 8? Superfans whose feelings you hurt by giving the game they are going to buy anyway a 7?
Before we do any more debating about the merits of putting a number on a score I want to know who cares.
Have any of you come across gamers who won't buy a game they were curious about because you gave it a 7 and not a 9? And, if so, would they have made a different decision if your review didn't include a score? Was it review scores that did in Too Human or put Wii Music slow out the gate?
Do scores ever really hurt or help games? Or are we just debating the best way to describe a game's quality, be it through numbers, words or faces in various stages of excitement?
***
Leigh Alexander, Gamasutra/Sexy Videogameland/Variety: Backing Stephen on this. Scores do stress the reviewer, but the stress doesn't come from any source that really counts—excepting maybe the anxiety I get sometimes knowing people's jobs depend on, say, Metacritic, and hoping that I was as thorough and fair as I could possibly be.
Like some others who've chimed in, I write the text first, and then see what score the text supports. In a way, I'm not assigning the number as a be-all measure of the game itself, but as a shorthand for my evaluation, and I think that's the function for which scores are most useful—as Jeff said, a single value that makes sense from a macro viewpoint for people who don't read text. There are a lot of those people (which makes me wonder if we shouldn't be trying to create more accessible, readable text, but that's probably a whole 'nother issue).
Finally, because I talked about preconceptions, I just wanted to clarify that while I may sometimes go in with ideas, hopes, dare I say biases, I never go in with a number in mind. The number is the last thing I come up with.
***
Dan "Shoe" Hsu, Sore Thumbs: To Stephen: Yes, yes and yes. They all care. But in the end, that review and score aren’t for them anyways -- they’re for your “normal” audience—so it really shouldn’t matter how bunched up their panties get.
At EGM, we’ve had plenty of readers who told us they would not even consider a game purchase if the reviews didn’t average a certain score they had set in their minds. Now, this “certain score” is usually a moving target—higher if it’s a game that reader wasn’t originally interested in, lower if he already had that title on his Amazon wishlist. This overreliance and faith in this one rating (scored by someone you probably don’t know intimately well) may seem silly, but I’m absolutely with Jeff and John on this: Despite complaints from a vocal minority, the vast majority of readers really want that number, letter, or direction the thumb’s pointing. It’s ingrained in society and it’s pointless and stubborn to fight it. People don’t always have time to read a 2000-word, well-crafted review to get inside the brain of the reviewer. For most folks in this short-attention-span world, that “4 out of 10” usually says more than enough.
Are we off-topic here, by the way? To answer question #2 above, yes, I sometimes change my score after I write the text. I do it like Robert initially: I score with my gut. But then while writing a review, I get to reflect upon my play time, think back to my progress 10, 20, or 30 hours ago, check back on my notes from last week, etc....and then I might adjust my score (usually by one increment up or down) based on my experience in its entirety. This may seem obvious—to score the whole game, not just how it finishes -- but I also have that same short-attention-span problem. And this helps me keep too much emotion out of the process, too, so a game’s high-note finish doesn’t unduly inflate the score.
***
Francesca Reyes, OXM: I can’t tell you how much I personally hate coming up with scores. It’s putting a quantitative label on something that’s qualitative. But that said, it’s a necessary and completely understandable function of game reviews in enthusiast pubs. (Hell, when I read movie reviews or book reviews, I do the same thing—look at the score to justify whatever half-ass, cobbled together pre-opinion I may or may not publicly admit to having.) The reviews I’m used to writing aren’t criticism in the sense that you’re a “reader” and you’re “reading” a game like you would text, or a movement, or even a movie—they’re practical forms of consumer advice. So, I agree with Jeff in that we’re here to inform our audience if a game is worth X amount of money based on whether it delivers, and to what extent, on its back-of-the-box promises, as well as how it handles as a game. And that demands a score, really. A signpost for what the review contains.
I’ve been lucky that all the pubs I’ve worked for fall in the realm of “enthusiast,” so in a lot of ways, you’re writing for gamers like yourself when you score a product. You’re also using a set of established criteria, depending on your publication’s ranking scale. If a 7 in a certain magazine means “average,” while a 5/5 stars in another means “must-have,” etc., this is what you’re working off of.
I remember writing a review of one game across multiple publications. One had a 100-point scale, another had a four-star scale, and yet another had something else. The thing that sticks out in my mind is that the 100 point one was for Ultra Gameplayers and the star-scale was Next Generation. UGP was your regular “review as a gamer.” Next Gen was “review for innovation and uniqueness, like ‘does this push the boundaries of its genre, etc.?’”. One game, reviewed against two very different sets of criteria. Interesting contrast.
So yeah, it’s totally ideal and utopian to think that you can sum up a game in a one-word or one-sentence definition from a pre-existing list of rankings, but this ain’t science. It’s voodoo magic in a lot of ways, no matter how hard we try to justify what the numbers, letters, or stars mean.
I try not to go into a review with a preconceived score in my head, but like Shawn said early in the thread—the result of months of pre-release hype or non-hype may or may not play a role in expectations from the reviewer and the reader. You may never really know. It’s just human nature, really, and publishers know this. But the trap is that we all play a shit ton of games, right? But our readers do not. Yeah, some of them play a lot of games that we haven’t. Some of them may play as many as or more than we have. But that’s the small portion of our audiences. Most of them pick and choose what they buy and we have to understand that spending their cash may rely heavily on what reviewers say. You have to respect that and go in to a game with the same expectations that someone without the months of exposure to a title might have.
Maybe that’s idealistic to expect this of writers who are supposed to be “experts” on their field (how do you become an expert in a field or medium if you’re not exposed to everything it has to offer, right?), but when it comes to reviews—in a lot of ways the boss is your reader and you have to kind of get in their skin. Am I always successful at doing this? Hell no. But it’s what I always aim for.
As for the actual process of coming up with a score—sometimes you just know, based on games you played before or a gut feeling when you’re playing it. I sometimes wait to put the score in the review until after I’ve written it so I can step back and get some perspective. If I have the luxury of time (shyeah), then I can let it sit for a bit and return to it for another pass to see if the score still holds. I like how Shoe mentioned the “high-note finish” and I agree. Games are experiences and once you see the entire narrative a developer has to tell you, there’s a sense of accomplishment that sometimes make you review through rose-colored glasses.
***
N'Gai Croal, Level Up/Newsweek: Robert, I’ll raise my hand and say that I do advocate putting an end to scoring. You wrote, "Scores seem to be the one thing that today's online audience can easily form a conversation around, and I think, ideally, a review should be like a conversation between reader and critic." Yes, a review should be a conversation between reader and writer. But /what kind/ of conversation do review scores foster? Judging by the Metal Gear Solid 4 example Kieron cited above, not much. (We’ll get into that more when we tackle Reader Backlash.)
Jeff, you wrote, "scores are primarily there to serve as shorthand for folks that won't or can't read the full review." Unless I'm missing something, that's quite an indictment of a portion of your audience. Why would you write for people who won't or can't read an entire review? You also say that scores "assist average, everyday people in their purchasing decisions." I don’t object to heds and deks; pros and cons; bullet points; final words; buy, try, fry; and other forms of giving readers a succinct take on a reviewer’s opinion about a game’s value. Heck, you could just tell them how much you think the game is worth. And like you, Fran, I firmly support the consumer guide function of game reviews. But anyone who won't or can't read an entire review isn't making an informed decision by looking at a single review score, so I’m not convinced that the reason review scores are “necessary” is to genuinely inform those consumers who can only focus on a letter grade, a number or a star rating.
In a different context above, Shawn talked about intentional fallacy. I think that's what's happening with review scores in the age of the Internet. Those of you who assign scores intend them to perform a certain function, but in the real world, the use to which they're being put by the most vocal portion of your readership is pernicious. Scores help bring out the worst in readers. They shut down conversations; foster silly debates; and they encourage meaningless comparisons. For too many readers, the very presence of scores turns the text of a review into a sideshow for the main event: this number, those stars, the orientation of that thumb. The text becomes a caption and the score becomes the photograph, en route to becoming the final cog in the Gamer Metrics-Metacritic-GameStop machine. Why would any of us want to perpetuate that?
This may seem easy for me to say from the perch of a blog and a magazine that isn’t dependent on gamers for its survival. But many of us work for or were previously employed by outlets that have been struggling. None of us are safe. Newsweek had a round of buyouts this year, the third in my nearly 14-year tenure at the magazine. Time Inc recently had layoffs. MTV had layoffs last week. Ziff-Davis closed Games For Windows magazine. The Tribune Company just declared bankruptcy. For years, magazines and newspapers have been moving towards bigger photos, more charts and even “charticles.” The Associated Press has imposed a 500-word limit on its entertainment writers. All of this devalues the importance of the word. Review scores are yet another signal to your readers that your words don’t matter.
If we don’t think scores are genuinely meaningful—there may be a robust defense of the inherent value of review scores; of the 6.5 versus the 7.0 and the 82 as compared to the 89, but no one has offered it yet—why do we continue this charade? Shoe, you wrote that the desire for scores is “ingrained in society and it’s pointless and stubborn to fight it” and that “For most folks in this short-attention-span world, that ‘4 out of 10’ usually says more than enough.” Apparently it’s not enough for most folks that we slit our own throats; we’re expected to provide the knives as well. I won’t pretend that yanking review scores will bring an age of genteel conversation or Socratic debate to the intertubes. But if message boards must be clogged with pointless argument, I’d rather it be fuelled by the words you wrote rather than the numbers you assigned.
***
Kieron Gillen, Rock, Paper, Shotgun: Since you ask for a justification for scores and the fetishized graduations of hundred-points scale, N'Gai, let me give it try. I'm from the wanky English tradition, and was anti-scores for most of my adolescence for the obvious reasons. They're stupid and putting any kind of scale over a subjective experience was ridiculous. For game reviews, at the current point in the medium, I've come around to them on solely utilitarian grounds.
For scores, per se: reviews exist fundamentally as buying guides, the proverbial shit-filter. They can do other things, but the reason to exist is to spend other people's money. When I score, for the pure consumer magazine (as opposed to one with a more critical leaning) I score for one thing only: should you buy it. For those less-prominent games, a high score is a direct tool to even make them read it - World of Goo and Braid getting a string of 10s is an obvious flare for ATTENTION! in a way which just giving them a glowing review isn't. If reviews are a shit filter, the mark is a very blunt tool for achieving that. That, to many gamers, is just what they want. In a real way, N'Gai, that's the job.
(This will come up later, but the thing about classical game reviews is that it isn't like film or music reviewing, but a hybrid between some stuff that's purely subjective and some stuff that's objective. You don't get a band's new CD which doesn't work on most CD players. You don't get movies which freeze randomly. For a review, that stuff genuinely matters and we're betraying our readers and being deeply disingenuous if we pretend otherwise. Are there other forms of games writing for other readers? Hell, yeah. But dismantling the review isn't the solution. We should just go and build something else.)
And the hundred-point scale? Perversely, what I most like about it is actually its weakness. It's inherently ludicrous. Who can tell the difference between 83 and 86 percent? No-one. In other words, its subjectivity is totally clear. The 10 or 5 point scale has a way of actually tricking people into thinking there's some science at work. The hundred-point scale--and calling it "percentage" scale is another thing that's deeply deceptive—is very silly. The fact it's a rough tool rather than a scientific implement is blatant. If you try to argue a few percentage scores you look as if you're suffering from some obsessive-compulsive disorder in a way you don't necessarily look like if you try to argue between a 7 and a 10. 100-point scales, treated correctly, are an ideal way to both act as a shorthand for the review, and simultaneously make it clear the mark isn't the review.
***
Shawn Elliott, 2K Boston: To add to N'Gai's thoughts....
Some of us suggest that our audiences sees scores as buyers' advice. Actual sales rarely correlate with review scores in cases where games are not also heavily hyped and marketed. Increasingly, gamers pre-order games prior to the publication of reviews. Interactive demos allow our audiences to decide for themselves whether or not a game will be worth their dollars. In addition, word of mouth and message board discussions inform our potential audiences' purchasing decisions with an intimacy and directness that we cannot provide. Finally, review aggregation sites such as Metacritic mute the bias of individual reviewers and provide a bigger picture.
I suspect these circumstances suggest that our self-perception is, well—a throwback to a time when magazines and websites were gaming's gatekeepers.
And yet we have John's anecdote about the angry reactions of some Computer Gaming World readers when the magazine dropped its scores. Robert introduced the idea that "review scores have one use: driving traffic from message boards and social networks to your site and giving those people an excuse to argue out their fan beefs in the comments section." Jeff countered that ratings "are primarily there to serve as shorthand for folks that won't or can't read the full review," which prompted N'Gai to ask why anyone would want to write for such an audience. Maybe our audiences aren't a homogeneous monolith—not in the sense that different readers look to a Level Up or Gamespot for different reasons, but that we (and our bottomlines) want or need different readers to look to a single site or magazine for many reasons. Is this part of the problem?
***
Stephen Totilo, MTV News: Shawn concluded his note with "Is this part of the problem?" I ask again, as I did in my previous note (with slightly different phrasing), what's the problem? Who or what are review scores hurting? N'Gai makes a passionate argument against the damage he sees review scores doing to the discourse about games on some message boards and comment threads. And he pitches a convincing case that such damage obscures the value of the scorers' words. All told, though, that doesn't seem like a whole lot of pain.
But, again, what's the problem? Are quality games not being appreciated because of the existence of review scores? Are quality critics not being read because of the proliferation of scores? Are talented game creators losing their jobs because of review scores? If yes to any of those questions, then would the abolition of scores remedy those situations? If not, I see no more reason for Giant Bomb and IGN to ditch scores than I see them needing to have their reviewers append to their reviews drawings of whatever flower the game they just played makes them think of. Whatever info the readers find useful and edifying, you know?
To the score haters, though, I direct you to Kotaku's reviews for support to your arguments. The mad bloggers there found a way to write reviews that don't use scores but can still somehow be comprehended in the time it takes to tie one's shoes. See their Far Cry 2 review, and, aside from too many puns, it gets the job done:
http://kotaku.com/5071946/far-cry-2-review-hurry-boy-its-waiting-there-for-you***
John Davison, What They Play: Kieron's point that game reviews have to be a hybrid between purely subjective commentary and some stuff that's objective is an extremely important one. The innovation and creativity of games design (or lack thereof) is more than just artistic expression, and an assessment of the overall experience is often incomplete without some mention of the mechanics. Criticism of games reviews often focuses on the fact that we spend too much time on this stuff, but it is often a more important consideration than the "art". I think we'd all love to spend more time on digging into what a game is trying to "say"—but we're often faced with impenetrable control mechanisms, or distracting technological problems that cannot go unmentioned. There are also still an awful lot of games, particularly in the current climate of "casual" style games on DS and Wii, that are practically *all* mechanics, and an intelligent discussion of how well they work is all that's needed.
Take a game like Prince of Persia, though. While there's much to be said about the art style, the way the narrative unfolds, and the co-dependence of the lead characters—a review of the "experience" cannot ignore the fact that the game has some nuts and bolts mechanical issues. It has a tendency to play itself, for a start. The controls are simplified to the point that it can be distracting to an experienced player, and that there are some unavoidable issues with the camera. They're boring topics compared to something that, say, focuses on an emotional response to the player's growing bond with Elika, but I think there's an expectation from our audiences for us to convey and assess the overall experience, not just the "heart."
Shawn, your point about the "homogeneous monolith" is a really important one. Times *have* changed since the days of magazines and websites being gaming's gatekeepers, and while we all tend to identify with our own outlets, and maintaining a sense of community that is distinct, we all have a part to play in the broader scheme of things. Given that Google is the window through which the world views the vast majority of content, there's less and less loyalty to specific outlets, and instead people are simply looking for opinions on specific topics. For this growing audience, a score is still important for exactly the reasons that Robert and Jeff have mentioned. It's shorthand. It's a hook. It's a way to get people to look at our reviews. We are all, after all, businesses. We need traffic. We need people to come to our sites and read our stuff.
N'Gai, you asked "Why would you write for people who won't or can't read an entire review?" I think that's oversimplifying. Often, our audiences (both the hardcore, and the not-so-hardcore) go trawling for review scores purely for validation of their taste. An abundance of high scores serves that purpose, and makes them feel vindicated whether it's because they've made a pre-order, or are simply fans of the franchise.
***
Robert Ashley, freelancer: I wouldn't want to separate "art" from game mechanics and interactive feel. Atmosphere and storytelling have been a fixation for games critics lately, but the connection between the player and the machine—the mysteriously engrossing feeling that makes people scrunch their faces up and open their mouths like zombies—is the real "art" of gaming. Technology and all its various failures play a huge role in that feeling. Anyone can tell when something isn't running smoothly, but I wouldn't call these observations entirely objective. Badly behaving cameras can drive some people insane. Other people not so much. The same thing applies to many common technical problems. The question is, did you have that great zen feeling of being inside and part of the game, or did technical problems and poor interface constantly break the spell? Making a game feel like Super Mario Galaxy isn't a simple question of technology and time. There's something special going on there.
***
Tom Chick, freelancer: On the subject of review scores and expectations, I have a lot on my mind when I review a game. I'm crammed full of preconceptions, expectations, prejudices, hopes, and fears. I call it "context" and it's probably the single most valuable thing I have to offer as a guy who writes about games. I've got thirty years of it under my belt. I don't let loose of it when I play a game, and I certainly don't let loose of it when I write about a game. I write as a hobbyist, and I write mostly for other hobbyists. We know how Spore was hyped, we're aware that Haze was from the Timesplitters guys, we've heard about the problems with Killzone, and we remember Trespasser. It's how we talk. When I have my druthers (i.e. when I'm not writing for a wider audience), it's exactly the sort of stuff that goes into what I write. And, yeah, it figures into whatever number or letter I have to slap onto a review. Like any gamer worth his salt, I have the bias of experience.
As for how much and when I think about scores, I think about them all the time. I think about how much I hate them and how much damage they do to the state of videogame discourse. Scores are an end run around saying anything meaningful. I hate when someone says (almost always on the internet), "I liked your review, but I would have given it an 8 instead of a 7". Because that's an unborn conversation that will never happen. If I didn't have to come up with that insufferable 7, the comment would have had to go as follows: "I liked your review, but I disagree with what you said about it being too hard" or "I liked your review, but I disagree with what you said about the graphics being too much like Fable" or "I liked your review but I disagree with what you said about the ending feeling out of place" or even "I liked your review, but I liked the game more than you did". Those are all starters for at least a line of thought and at best a conversation, and in either instance, we can both be the wiser for it.
Review scores are for the lazy, the unengaged, and the inarticulate. They're for stickers on boxes and press releases. They're understood differently by different people, and they're applied differently by different publications. They're an attempt to inject some sort of science into someplace it doesn't belong and the sad irony is that they mean nothing. I don't know if games are art, but so long as we're branding numbers into their flanks, they're certainly consumer products.
Now I have the luxury of saying all this, because unlike some of you (well, John and Dan, at one point), I don't have to run a magazine or website. To folks dealing with lazy, unengaged, and inarticulate readers, I don't envy you your job selling stuff to them.
***
N'Gai Croal, Level Up/Newsweek: Harry, you came up as a journalist and critic alongside hip-hop. You wrote for The Source, with its famous one-to-five-mics rating system and all the arguments—inside the magazine and among its readers--that were stemmed from its reviews. You also worked for the Village Voice, an outlet that didn't apply scores to any of its reviews—with the notable exception of Robert Christgau's monthly Consumer Guide, which introduced letter grades to music criticism—yet fueled legendary debates nonetheless. You worked for Rockstar Games. You're even writing a book about architecture in computer and videogames. (I bow.)
What do you make of all of this? There's nothing new under the sun, so you must have been in or around discussions like the one we're having right here—does it bring back any memories you can share? Are Jeff and Francesca correct when they say that scores are a legitimate part of consumer reviews, regardless of how some readers may respond to them? Are Tom and I right to argue that review scores help engender the all-too-often juvenile discourse that surrounds videogames? Is Stephen onto something when he suggests that the case against scores is tenuous, and that as long as some readers find scores edifying, reviewers who choose to score games shouldn't agonize terribly over doing so? Is John right that Rolling Stone's editors, not its writers, assign the star ratings? (I had to try, man.) And finally, what's it like being on the receiving end, watching scores and reviews trickle in, and—tell the truth, now--which mattered more?
***
Shawn Elliott, 2K Boston: Before we finish this section with a final question on the real consequences that review scores can carry (an elliptical response to Stephen's repeated "What's the problem?"), I want to take a moment to address any unanswered inquiries. Note that, for the moment, I'm withholding my thoughts on the intersection of mechanics, meaning, and the stories that games try to tell (the short, unsubstantiated version is that I agree with Robert that it's largely fruitless to look at these in isolation).
Kieron commented that he thinks "AAA-popular-sequels tend to start with 9/10 and lose marks, while games with less expectations start with 5/10 and have to gain them." This strikes me as especially true for enthusiasts. Anecdotally, it also seems as though critics are more inclined to take the gloves off with less-anticipated, lightly marketed games or, conversely, to forgive their faults. In addition, Leigh wondered about the wisdom in bringing an awareness of a game's budget to our analyses. Thoughts?
And then there's John's Rolling Stone rumor. What do we make of a critic submitting copy and his or her editor supplying the score? It's an interesting thought experiment. I do think that the policy would encourage writers to explain themselves more carefully, and that it might erase much of the discrepancy in words and numbers, but what ramifications would it carry?
***
Jeff Gerstmann, Giant Bomb: I find Kieron's comment about big budget games catching a bit of a break in reviews to be possible in isolated cases, but I don't think it's the norm at most publications. I've also seen the opposite, where the big budget game gets trashed for not living up to the insurmountable mountain of marketing hype while the low-budget indie darling catches a break because it was made by a team of five people or something. And I've certainly seen little, unmarketed games get absolutely thrashed in reviews. Sometimes it seems like the reviewer is doing this because it's "safe" to do so, like it was part of some sort of "see? We totally use our entire 1-10 scale" chest pounding. This is why it's important for reviewers to have an editor (or editors) that can keep them in check and ask questions about a review and its score before it gets published, especially at outlets where that one review is meant to represent the entire publication's view on a game.
While I've never been a party to anything quite like this Rolling Stone rumor, I've spent a great deal of time as an assigning editor for reviews. The most important part of that position is working with the authors on clarity, to make sure they actually mean what they say, and that they aren't coming off as more positive or negative than they intend. Sometimes that involves changing the score to make it match the words more closely. This would happen most often with freelancers, as they can't really be expected to be experts of how one publication's scoring system differs from another. But except in extremely rare cases where the reviewer wasn't available at post time, those changes were made after discussing it both with the internal staff (as part of a review vetting process) and with the author. I wouldn't want to do things the way Rolling Stone supposedly does them, but that might say more about my faith in most freelancers than it does about the policy itself.
Lastly, I really don't think a game's budget matters when reviewing a game. High-budget console games and mid-budget console games cost the same $60. The only dollar amount that matters is the retail price. While bad games certainly don't get dramatically better as their retail price drops, it's a lot easier to overlook some of a game's flaws if you're getting it for $20 instead of $60. But at the same time, it's probably fine to mention a game's budget in passing. There's a big difference between a brief mention of the budget and using it as the centerpiece of your entire article.
***
Leigh Alexander, Gamasutra/Sexy Videogameland/Variety: I'm super eager for Shawn's discussion point on the impact of the scores, and I don't want to drag everyone down a side trail, but wanted to note something on the Rolling Stone rumor.
No idea what they do at Rolling Stone, but I've actually been in situations more than once where the final star ranking or number was suggested by my editor. It's not as sinister as one might imagine—it isn't as if a person who didn't play the game is independently applying a number with no input from me. In fact, it's more of a collaborative review process between the editor and the writer to be sure that the final rating really does correlate to the text as it's written. Limits in what you can express within a word count, which I'm sure Rolling Stone is constrained by, can make it useful to have two pairs of eyes on the situation.
In fact—and I think this is especially true for those who write for more mainstream print—many reviewers in that context want to avoid the wham-boom core market number hysteria. They want to write articles, they prefer to write crit, and don't want to calculate numerics. They may be non-traditional reviewers (as I'd assume Rolling Stone's are). In that case, especially where Metacritic is involved, in my experience an editor may volunteer to apply a score that's correlative with the review text simply to offer an option for those writers who don't want anything to do with the numbers game.
In those cases I have always elected to have input because I feel comfortable that way—but keep in mind that scores can cause headaches and PR arguments for reviewers, and in that case, an editor stepping up and saying "point them to me if they hate the number" may be one way of allowing the writer to do his or her own most honest work with impunity. When an editor plays a role in the score, he or she's essentially "backing" the writer's text by shouldering all those burdens a number can provoke. It's a pleasant reversal from the nightmare scenarios we hear about where writers catch flack from their bosses or even get canned because some publisher flipped their lid.
Again, I don't even know anyone who writes for Rolling Stone. I wish I did, because I'd tell them to write about good music again. But my suspicion is that scenario doesn't so much involve some sinister overlord stealing scoring power from the reviewer, rather an editor who wants to make room for the writer to do what the writer's being paid to do—write well and thoroughly on a title.
***
Shawn Elliott, 2K Boston: The editor in me empathizes with Jeff, especially in cases where a green writer's copy is in apparent conflict with the rating he or she attaches to it. Discussion is helpful here, as the writer may reveal anecdotes and a level of analysis lacking in their initial draft. How many of us remember the teachers who told us they wanted to read in our revisions whatever it was we'd just said?
The writer in me, however, is just as wary of knee-jerk editing, and editors who've played -- and have formed their own opinions of -- the games they've assigned to freelancers. Here
http://www.gamesradar.com/pc/call-of-duty-4-modern-warfare/review/call-of-duty-4-modern-warfare/a-2007110512090683184815/g-20070214121215902085 , for example, we find text apparently taken from PC Gamer UK's Call of Duty 4 review (
http://www.computerandvideogames.com/article.php?id=175050&site=pcg) transported to GamesRadar's 360 review page where a presumably miserly British 8.5 becomes a prodigal American 10. Bylines differ, too, as though we're to believe that two writers, an ocean apart, have arrived at the same sequence of words and altogether different assessments of their meaning (the editor's note states that the GamesRadar score was once a 9, which still doesn't explain discrepancy). That Francis's copy came first is fairly obvious in lines like, "If the whole game had been like that, or even just as inventive throughout, you'd find a frankly silly score at the end of this review. Instead it's a more restrained one[....]" My suspicion is that a comatose editor couldn't be bothered to read let alone edit the original. Frankly, that's a fucking insult to writers and readers if there ever was one. It says, 'Hey, fill some space for the cretins naive enough to not notice. We'll handle the hard work."
Hopefully, this is an anomaly. The idea of dismissing the Rolling Stone idea for it is as dumb as it would be to eliminate governorship because of Blagojevich, but, again, we're all human.
***
N'Gai Croal, Level Up/Newsweek: Regarding the Rolling Stone rumor, I'm think it's legitimate for reviewers and editors to discuss scores before assigning them, but I'm wary of editors assigning scores all by their lonesome. This point was driven home for me when, in the wake of Jeff's firing from GameSpot following his Kane & Lynch review (
http://blog.newsweek.com/blogs/levelup/archive/2007/12/05/reflections-on-videogame-publisher-and-employer-contempt-towards-the-enthusiast-press.aspx), Tor Thorson issued a statement explaining that "The copy was adjusted several days following its publication so that it better meshed with its score, which remained unchanged." (
http://www.joystiq.com/2007/12/05/gamespot-addresses-gerstmann-gate-concerns-in-depth/) Even given the force majeure of this situation, the question that immediately popped into my head was this: if changes had to be made at all, why wasn't the score changed to better mesh with the copy? If there are going to be scores, the individual reviewer should always be intimately involved in that process.
*****
Shawn Elliott, 2K Boston: To finish this section and finally return to Stephen's point that scores, despite 'damaging the discourse about games' and 'obscuring the value of words,' aren't an actual problem, I'd like to ask one last question.
Review writing carries real consequence because some publishers do base developer and PR bonus pay on aggregated ratings. This shouldn't concern critics, but once-warm PR people and game producers can become cold upon our publication of undesirable review scores, diminishing or eliminating our ability to secure subsequent interviews and access. Postmortem discussions and exclusive looks at the publisher and/or developer's forthcoming products are less likely. Conversely, a few publishers will permit us to post reviews before competitors, provided our review scores are favorable. Do such pressures produce a subliminal background—especially among members of the enthusiast press—or even enter our thoughts as we write or edit reviews and assign reviewers or scores? The stock answer says, "Only if you're a bad apple, and I'm not." But isn't the seeming impropriety of business in a bad barrel a problem in itself?
Consider special instances such as the Gamespot Kane and Lynch review that N'Gai cites. Jeff, are you contractually able to discuss that episode in any detail? John, Francesca, and Dan, as well, serve/served as EICs of enthusiast publications and presumably face/faced such pressures and repercussions.
***
Dan Hsu, Sore Thumbs Blog: Exclusive reviews stink...it’s a lose-lose situation. If you score high, then readers call shenanigans...even if there were none. If you score low, then the game makers get mad and won’t want to work with you in the future.
Plus, most companies want a guaranteed score or range of scores. We did play with this fire once at EGM...the very first time I was faced with it. I thought I could deal with it by protecting the reviewers from that discussion—I’d let them review the game independently and then see if the scores were high enough for us to secure that exclusive. But because I had final say on all magazine content, I’m still a part of the reviews process. In the end, I didn’t feel comfortable promising any certain scores, so I backed off and made a rule to never entertain these offers again. If we don’t even have those discussions to begin with, then we’ll have preserved the integrity of the reviews from start to finish.
Besides...if the reviewer caught wind of the deal, how can it not spoil things, however so slightly? Let’s say the reviewer’s 50-50 wavering between an 8.5 and a 9.0, but way back in the deep recesses of his mind, he knows that a 9.0 will get his website or magazine the exclusive review, which translates into revenue-producing traffic or sales. Will that load the dice? Maybe not for everyone, but that’s not the point. Once that process is tainted, it’s tainted. Just ask the reader what he or she thinks.
***
Leigh Alexander, Gamasutra/Sexy Videogameland/Variety: I think in Jeff Gerstmann's case, the fact that the editors took liberty with the final text and scores are evidence of the "Rolling Stone System" not working in practice—again, I support the idea of editors protecting their writers (as Shoe tried to do with his folks, it seems), and doing whatever they can do to keep the heat of publisher response to scores out of the way so that the writer can do his best work. The "protection" obviously didn't take place with Jeff, and while I've no inside knowledge of that situation it sure looks to me like they let him take a bullet, which is at the very least evidence that making any kind of bargain with a publisher regarding a review—its exclusivity or otherwise—is a BS proposition.
I hope that this reason, coupled with the evolution of Internet media, means that exclusive reviews are going to go the way of the dinosaur (or of the console exclusive, har har). I wholly believe we now work in an environment where quality and depth can be the primary competitive advantage, and less timeliness.
I also can't speak for Totilo, and don't mean to put words in his mouth, but when I agreed heartily with his response, I was agreeing with the idea that scores only cause problems for the industry and for us—neither of whom is the audience we serve, and therefore that they create stress is not a compelling argument against the fact that plenty of readers find them useful. Hell, I hate scoring as much as most of us do, and I prefer scoreless criticism and blah blah blah—but in all honesty, when I really want to know whether I should pay attention to a game, the very first thing I do is go to GameSpot and see what number they gave it, for whatever it's worth.
I never buy based on a number (let's take bets on how many—or how few—year-end top ten lists the 10-rated MGS4 or GTA IV get), and I don't think readers do either, which further takes the piss out of this idea of their damages. Believe me, I'd love to stop scoring. I'd love if we could enforce an industry-wide moratorium on scores, so that we wouldn't have to think about 'em and so that our audience could re-learn to focus on the words we write.
But I think we hate scores because of the undue importance that has been placed on them and the ways they've distracted from our work, and this is credited in part to editors balancing the needs of the publishers with their management of irrationally hostile reader reactions, neither of which should be their focus. None of this means they in and of themselves cause harm to the readership or to the art of the review.
***
Shawn Elliott, 2K Boston: Dan, do you see where this can apply to cases less extreme than exclusive reviews? Let's say that a publisher puts one game in the market this month and is also looking for an outlet to announce a forthcoming game a few months from now. No conspiratorial conversations or conniving occurs. Nonetheless, you have to know that by giving the available game a negative review, you could risk reducing the likelihood that the company will allow you to reveal their other title. Scruples aside, the cost-benefit equation is simple: A first look at the unannouced game is probably better for business than the bad review, provided nobody calls bullshit. And compromise comes in shades. An EIC can always assign the review to a specific staff writer who he suspects will appreciate the game more than his peers, and argue afterwards that he hadn't touched the score itself. I have no idea whether or not this happens. I do, however, know that commercial publishing is cutthroat.
***
N’Gai Croal, Level Up/Newsweek: From the outside looking in, I’ve been conditioned to expect that review cover = exclusive review = 8 or above. That led to a bizarre reading experience last holiday when I picked up a copy of Official Xbox Magazine with Turok on the cover, touting the first review. I can’t lie, I immediately flipped to the end of the review to see what score had been handed out…only to be shocked to see that it had been tagged with a 7! What went through my mind next was:
a) Damn, that took guts for Francesca to stick to her guns and let reviewer Paul Curthoys call it like he saw it.
b) there goes Christmas for Josh Holmes and the rest of the guys over at Propaganda Games, to say nothing of Graham Hopper and the folks at Disney Interactive.
c) I would not want to be the publicist who brokered that exclusive. Her/his job is to suss out the most favorable outlet when determining who’s going to get that first review, because you only get one chance to make a first impression. But a first review score of 7 means that something went terribly wrong—and from PR and marketing’s perspective, the “something went terribly wrong” in this kind of situation is /not /the finished game’s quality.
At last glance, the Metacritic rating for the Xbox 360 version of Turok was 69, so OXM’s review was right in line with the critical consensus as calculated by that aggregation engine. I’m sure that was little consolation for Propaganda and Disney, though.
John, didn't you write about the pressure surrounding review exclusives when you first started blogging?
***
Dan “Shoe” Hsu, Sore Thumbs: Shawn, that’s why this business is so messed up! Publishers want good reviews. Editors want exclusives. Magazines and websites want advertising. Advertisers want good reviews.
Sometimes, the companies know their products aren’t that great. They have internal research (via their own gut checks or “mock reviews”—early evaluations done by outside consultants) that give them a rough idea of how a game may score. If you’re dealing with a reasonable PR team and you’ve scored within their expected range, then you’re usually spared the grief. But woe is the outlet who’s at the bottom of the Game Rankings or Metacritic lists. We’ve had PR people complain to us that we’re “outside the average” (on the low end) on those sites, which John and I would always laugh at. Of course someone has to be outside of the average. If everyone was ON the average, there’d be no point in averaging them!
***
Jeff Gerstmann, Giant Bomb: Exclusives are yet another by-product of this business' print roots. Between magazines stories getting scanned, website text getting pasted into message boards, and the way that most of a site's traffic doesn't come in through the front door (where the big promotion of the exclusive always is), exclusive stories are waaaaay more trouble than they're worth. Exclusive reviews doubly so. On top of that, in my experience, most big exclusives don't really move the needle that much, traffic-wise. They've always felt like a huge waste of time to me. Attracting users by differentiating your coverage from the pack has always seemed like a better goal.
But are we talking about a problem that doesn't really exist anymore? Isn't IGN the only publication still doing exclusive reviews? And if so, they're probably getting them because they're the only ones asking for them, right? I think they've deflected questions about their integrity with the ol' "well, we're not on the take, case closed" routine, which the majority of the public seems completely fine with.
***
Stephen Totilo, MTV News: Many of you were struggling to make a decent argument for the elimination of review scores. Here, I think you just did. Don't put scores on your exclusive reviews in your magazines. Add them via your website two weeks later when everyone else's reviews run. Then everyone's happy.
Or…just don't take exclusive reviews. This would be a challenge for a magazine that needs to secure something notable months in advance of press time. But for websites, is there really a need? You can get the game when it comes out, play through it and have a review up within a week of it's release. No tricky politics. No stink of compromised values. And your readers are served.
The threat of publishers withholding cooperation in the future because of a review strikes me as absurd. I don't doubt that it's a real threat, but there are countless ways to report and edit around any blackballing. Besides, getting blackballed usually makes for a good story.
***
Robert Ashley, freelancer: Just to add to the list of reasons why reviews create editorial drama, it's my understanding that retailers take into account review scores when stocking new games, which means that scores can directly affect sales. I had a PR rep from Deep Silver tell me last week that Target bases their game orders exclusively on Gamespot scores (so, of course, he was bitching about all the snooty divas at Gamespot). I can't vouch for his claim, but just having the guy go off in a room full of other pro enthusiasts noticeably changed the mood of everyone in earshot.
While reviewing for EGM, I felt like there was always barrier between me and PR, thanks to the reviews editor. But when I would venture out to write other stories, visiting studios and going to game reveal events, I would often get snippy comments from people about my reviews. I was once accosted by a developer from the True Crime: New York team about my review of his game. We had, by chance, ended up in the same two-person sleeper car on a train from Paris to Venice. And I was on a family vacation, not some gaming press junket.
***
Dan “Shoe” Hsu, Sore Thumbs: Stephen, getting that first review up the weekend before a game’s out versus a week after launch can be a huge, huge difference in traffic. So most website editors would find an exclusive review very appealing. Can’t blame them.
Those threats from publishers can come in many different forms. Sometimes, it’s just one very specific team or division that wants to blacklist you. For example, we were supposed to get an online exclusive as part of a package deal with our Saints Row 2 cover story, but after they read Robert’s article, THQ told us that the developers didn’t want to work with us anymore. So that online exclusive went to someone else instead.
The Mortal Kombat team and Sony’s sports division banned us as well. But the interesting thing is, these three blacklistings didn’t carry over to the rest of THQ, Midway, or Sony Computer Entertainment. They were very specific to those specific products, because they felt we had it out for them.
Then there are situations where the blacklistings go company-wide, like with Ubisoft after our Assassin’s Creed reviews...and after we wanted to do a story about how outlets were allowed to break the universal AC reviews embargo if they scored the game high enough. That “non-cooperation” on our part was the last straw for Ubi, and when I reported on this blacklisting in EGM, it just further cemented their hatred for me as an individual (even though Sam Kennedy, Patrick Klepek, and just about every other editor wanted to run those stories as well).
Now, I don’t really think Ubisoft or any of those other companies have done anything wrong here. It’s their right to work with whomever they want. But it just goes to show how uncomfortably and inappropriately cozy the industry and enthusiast press are expected to be. You cooperate, you benefit. Simple as that.
***
Kieron Gillen, Rock Paper Shotgun: *Yeah, Shawn, regarding exclusives, on the UK side it's pretty similar. On the print side, perhaps even more intensely. There's just more magazines and the competition has always been fiercer. The former is primarily due to the size of the country - the shipping costs don't cripple you, and a smaller magazine can be more profitable. The second is due to how British games magazines are sold. The vast majority of magazines are sold at newstand, not subscription. Subscriptions are cheaper, but not a straight loss leader for ads. When I was on Gamer, the majority of our money came from actual people buying the bloody thing rather than via advertisers. On the bright side, this abstractly means that in a real way, the bills are paid by people who want to read your mag. On the bad side, since there's an enormous floating readership which you have to fight for every single issue, meaning even more importance is placed on exclusives. An enormous amount of effort is thrown into chasing them, and it can totally lead to the sort of issues others have picked up on.
When I'm chatting to modern day magazine staff, one of my favorite stories to tell is about an old pre-me Future major-mag editor. He didn't speak to any PRs. The Dep Ed did all of that. If any of them tried to speak to him, he just blanked them. He just made the magazine. And when I roll out that anecdote, the look of disbelief which it's always greeted with is akin to what I'd have got if I claimed he was capable of flight.
As I said, they're an enormous waste of energy—I always remember that it seemed that due to the inevitable pissing off of PRs, the exclusives we chased went to our major rival as much as not, and the ones that they went for went to us. But it's also a game which I think magazines—and publishers—are loathe to get out of it. Because as long as one other organ in the room is making exclusives, you are at a distinct disadvantage. And it works both ways. One major games publisher, working on similar logic to me above, stopped doing exclusives for a few year. The amount of coverage suffered—because their rivals were still making deals and they weren't—and they went back on it.
That's the problem with stopping exclusives. It requires a conspiracy of doves for it to hold.
Regarding changing marks and having marks changed... much like others have said, I've done it. It's really not a problem per se. Discussing it with the writer is fine (And an art form in and of itself). Explaining how they're not marking to your mark-scheme is fine too. And most commonly, the review and score just don't match up, where you have to ask that one or the other is changed. If a reviewer is having fun comparing part of the game's mechanics to cancer and still give it in the sixties, either the overwrought writing goes or—if they actually mean the overwroughtness—the mark needs to drop like a stone. In my experience, I've argued far more marks down than up.
On the other side, I'm fine with that too. Sometimes I really don't agree, in which case I ask for my name to be removed from the review. Not in a prima-donna way—just because I don't believe in it, and ultimately, your name is all you've got and you're going to give enough review scores the readers will tear you apart on your lonesome without someone else making them for you. This is one of the first things I say to any writer getting into the business, because—bless 'em—not many are even aware that it's something they can ask to do. And if the editor says no, it's about as good a sign that you should get out of that job sharpish.
And, of course, some editors do change marks without asking and keep your name on it. Just don't work for them again.
***
Harry Allen, Media Assassin: Greetings, everyone.
I just want to say that it's really a privilege to be able to read all of your thoughts. I'm a fan of you all. I read your words and listen to your podcasts, in many cases. So, it's great for me to hear your thoughtfully written ideas about this very trenchant aspect of your work.
I don't know that I have anything to add. Much of what I was thinking has been said by other people, and I know we're about to change topics. For example, a very early concern of mine was the relationship between scores and advertising, so I'm glad that some of that was addressed recently.
I will say that one of the first statements I read that directly connected to my own work was N'Gai's comparison of the reviewer and the critic. Though I use different language, I know that when I started writing about hip-hop professionally, in the late '80s, I made it my objective to never talk about an album in terms of whether I liked it or not. Instead, I always saw it as my job to explain the artist's intent to the readers. I've never called myself a "music critic." I've always said that I am a Hip-Hop Activist.
To me, number scores are mostly an attempt at giving the illusion of numerical precision to functions that cannot possibly bear the same.
For example, take a look at Robert Motherwell's 1961 painting, "Elegy to the Spanish Republic, 70," here: (
http://www.metmuseum.org/TOAH/HD/abex/ho_65.247.htm)
Is that a 10? An 8.5?
Most curators and art historians would call the bold canvas "a masterpiece," and leave it, more or less, at that. If they did quantify it, or one of the others in the series, they would only do so in extremely hushed whispers, among themselves, if at all, but never do so for public consumption. This is actually part of the way that they frame their field, which is commercial, but which is mostly a repository of serious, high culture: By avoiding quantifying numbers ("It's an 8"), which would be seen as crass.
Quantifying a Motherwell, then, would be analogous to asking how much the Maybach is: If you have to ask, you can't afford it. They're not saying you can't afford the Motherwell financially, but you "can't afford it" aesthetically. In other words, the experience of the painting is far too rich to reduce to a digit, and if you don't know that, you shouldn't be here.
I think a reason similar to this is also why I always resisted, during that brief period of my life, when male friends would ask me to assign a number to a girl I've seen. In hip-hop / Black slang, a "dime" is a girl who's a 10...but what does that mean? According to what objective scale?
And indeed, isn't that the core idea that disproves the fantasy: That without an actual 10 to which one can point—the theoretical perfect game—the numbers become meaningless?
That is, on a foot-long ruler, "4" only means something because there's a "7," and a "9," but, most of all, because there's a "12." However, "12" only means something because there's a "13" and "25"; an agreed-upon metric, in other words.
When it comes down to it, game numbers facilitate the purchasing decisions of the buying public, and advertising, and I think that's it. I could have told John Davison that his CGW readers would revolt. It's like all those Americans who claim they only watch PBS and the news, yet, somehow, reality shows are a phenomenon, and porn even more so.
But, clearly, the focus on numbers is leading people to overlook something important, namely the subtle interplay of parts that is a videogame.
I've not played Gears of War 2 yet, though I did have the nerve to write about the game's marketing on Media Assassin, here (
http://harryallen.info/?p=1863).
But when I look at stills and video from it, and hear descriptions of what Cliff Bleszinski and crew were trying to do, it's clear to me, Roger Ebert be damned, that we're dealing with a moment here. Gears is a work of art expressed in the videogame medium. It's that simple.
Here's a prediction: A hundred years from now, collectors will purchase copies of Gears, Katamari Damacy, System Shock, Crysis, Ico, Dead Space, and every Mario iteration the way many, today, buy coin banks, first edition books, fine watches, and music boxes.
I'm sorry to say this, but, by that point, those collectors will not care if these games were given sixes or nines. They will collect them as beautiful art expressions of human craft and intellect. For many of the artists who work on these games, it will only be then that, aside from the thoughtful critique of people like you, they get their just due.
###
