Thursday, December 18, 2008

Symposium Part One: Review Scores

Introduction
Are reviews primarily a consumer guide, or should they serve another purpose? Do review scores deter intelligent discussion of videogames? Is the presence or absence of a review score the only difference between a reviewer and a critic? What is the role of the reviewer when the Internet is democratizing published opinion? How should reviews and reviewers evolve in light of the emergence and growth of Flash games, small games, indie games and user-generated games?

These questions and more were on the mind of N'Gai Croal, John Davison and Shawn Elliott last summer when they decided to expand their conversation to a number of noted reviewers, writers, bloggers and journalists for a published email symposium on game reviews. (See below for the full list of participants.) The planned list of topics include Review Scores; Review Policy, Practice and Ethics; Reader Backlash; Reviews in the Age of Social media; Reviews in the Mainstream Media; Casual, Indie, and User-Generated Games; Reviews vs. Criticism; and Evolving the Review. Round 1's topic: Review Scores.

Participants

Leigh Alexander, Gamasutra/Sexy Videogameland/Variety


Harry Allen, Media Assassin

Robert Ashley, freelancer

Tom Chick, freelancer

N'Gai Croal, Level Up/Newsweek

John Davison, What They Play

Shawn Elliott, 2K Boston

Jeff Gerstmann, Giant Bomb

Kieron Gillen, Rock, Paper, Shotgun

Dan Hsu, Sore Thumbs Blog

Francesca Reyes, Official Xbox Magazine

Stephen Totilo, MTV News



Review Scores


Shawn Elliott, 2K Boston: How much is on our minds before we begin playing any given game for review purposes? Will we imagine a range of probable scores that a heavily marketed, highly budgeted, and hugely anticipated game will get? What when the game is branded “budget” or is the work of a lesser-known, less-storied studio? If so, how closely have actual scores correlated with our assumptions?

Kieron Gillen, Rock, Paper, Shotgun: As others have said before—but Troy Goodfellow put most snappily, so I'm stealing his phrasing—the games press has a presentist/futurist bias. The vast majority of press coverage is for games that either aren't available, or are only just available. Even if we haven't seen or played the game personally, our peers will have. And we'll have seen comments threads full of people saying what *they think* of the edited information of the game we (and their PR) have presented. And with all that, when you throw a score out you know it's going to be read with those expectations in mind. When Eurogamer's Metal Gear Solid 4 review gave it an 8/10 there were 2000-post threads and actual death-threats. And Oli [Welsh], when he wrote that review, knew exactly what response he could expect. Games without the hype have lower expectations. I remember the attitude being crystallized by a comment I saw ages ago on Kotaku which stuck with me, when they linked to a B-game someone had 9/10ed: "It can't be any good, as I haven't heard of it". It's an ugly, but common, tautology.

You can't avoid knowing what the score is on that point, without becoming a true hermit. In terms of coloring your actual expectations of the game per se... well, unless someone's actually paying me to research a feature, I ignore 95% of previews. So when reviews come up, I try to review what's there rather than the hype... but that's going onto a whole different question.

Quick thought regarding the indie/AAA dichotomy, though: I often think that AAA-popular-sequels tend to start with 9/10 and lose marks, while games with less expectations start with 5/10 and have to gain them. And... oh, I'll shut up. More on this later, I suspect.

***

Leigh Alexander, Gamasutra/Sexy Videogameland/Variety: So, as far as preconceptions go, I just thought it worth noting that a game's marketing machine, whether through its fierceness or its clumsiness, would very much for like for us to have a preconception going into a review.

Unfortunately for them, they can't necessarily pick what impression they create. I like to think we react to the fashion in which we're being messaged, rather than devouring piecemeal the messaging itself. Or, most of us do.

So, I agree with Kieron that the right answer is "no preconception"—i.e, the reviewing process doesn't begin until you start playing the full version of the game, period. But sometimes I wonder whether background factors should be considered as context for a review. For example, for months a hyperbolic individual promises that his game will revolutionize ludology. Are we allowed (or, conversely, obligated?) to consider his lofty goals when evaluating the end result? If a company creates an "identity" for a game ahead of time, shouldn't that exemplify what the game is aiming to be, and shouldn't we try and consider whether or not it achieves it?

There's a line, I think, between making a prejudgment, and bringing with you a context within which to make an evaluation. Games are an industry and a culture, not a fragmented, compartmentalized list of disparate products, and rather than pretend we have no early opinions, I wonder if it's not beneficial to be prepared to bring that context—which also applies, perhaps to being aware of budgets, of team sizes, of other challenges?

***

Shawn Elliott, 2K Boston: Because I believe that self-enhancing, self-serving, egocentric biases are normal, and that people are prone to see themselves as being immune to the influences that move everybody else, I'll happily admit—along with Kieron—that I have preconceptions before playing. I'm human.

I'd argue that our preconceptions are active when we decide which games we want to review. That's not to suggest that, when given the choice, all critics go straight for the gravy (I've often volunteered to review games that I imagined would be interesting but not the best available). But what, if not a preconception of some sort, drives these decisions?

In addition, I believe that my assumptions are active as I play. For instance, I'm less likely to immediately doubt the wisdom of a given design choice in a Valve game than I am with the work of second-rate studios. An analogy: Say you're competing against someone with sorry win-loss stats in a strategy game. His opening moves seem odd, so you assume he's stupid. When his record is intimidating, you take the time to study his seemingly odd tactics until you're certain you're not missing something. In my mind, "the right answer" isn't a realistic answer.

Leigh, I have a problem with holding a loud developer to his hyperbolic promises (and it has nothing to do with the dozens of programmers, designers, producers, artists, and animators hanging their heads behind him): intentional fallacy. I'm interested in the degree to which game maker's games match their ambitions, but I wouldn't want to evaluate them on this basis. What New Critics wrote of poems seems sensible for games: "It is detached from the author at birth and goes about the world beyond his power to intend about it or control it. The poem belongs to the public."

Should we consider budgets and staff sizes? Certainly not when the critic's intent is strictly to inform consumer shopping sprees.

***

N’Gai Croal, Level Up/Newsweek: I’ve never liked assigning scores as part of any critical assessment, and the times I’ve had to do so in the past, it’s always been under duress. I started out as a journalist by writing movie reviews for my college paper, and none of the critics after whom I tried to pattern myself—Pauline Kael, J. Hoberman, Stanley Kauffmann, John Simon, Andrew Sarris, Armond White—used stars or points or thumbs. They didn’t provide you with any shortcuts or shorthand. You had to read what they wrote in its entirety in order to figure out what they thought. I said to myself, when I grow up, that’s the kind of critic that I want to be. So because I’m not obligated to dole out review scores in print or online, I only have two things on my mind when I start playing a game that I know I’m going to write about.

First, am I going to enjoy this game? In that sense, it’s not dissimilar from when I take in a movie. Or a TV show. Or a play. Or a book. Even when it’s a shared experience, playing a game is intensely personal, and no matter the developer’s pedigree, no matter the budget, I start each new title the same way: on the precipice between hope and fear. I hope that it will be good or great. I fear that it will be mediocre or worse. And as I give myself over to that series of firsts—the first image, the first sound, those first bits of gameplay, that first accomplishment—any and all external influences evaporate, leaving me only the thrum of my internal gauge, the one that tells me just how much I’m enjoying myself. I trust that gauge implicitly, and while external factors might influence precisely how I /articulate/ my opinion, I don’t believe it goes much beyond that.

Second, how much of this game am I going to be able to complete before my deadline? That’s very different from how I approach plays, television, theater or literature—I wouldn’t dream of critically assessing a piece of work from those media without having completed it. Why doesn’t that stop me from doing the same with videogames?

The explanation—or is it an excuse?—that I offer is that I don’t review games. We’ll get into this more in the Reviews vs. Criticism section of our symposium, but the way I see it, a reviewer answers the question, how well does this game work, but a critic answers the question, how does this game work? A reviewer helps consumers decide whether or not they should buy a game; a critic helps players think about a game that they’ve played—in its entirety /or/ in part—and that is the end of the spectrum where I believe my writing lies. (That’s also why, on a game by game basis, I don’t think I need to have completed a game to have some insights about it—but I do think that if I were advising someone on how to spend their money, I’d feel obligated to play most or all of the game.) Scores can serve as a valid form of shorthand for the work of the reviewer, but I’m not convinced that scores have much to offer the work of the critic.

***

Kieron Gillen, Rock, Paper, Shotgun: Leigh, I agree with Shawn. You can mention the hyped intention and mention whether it measures up—but that's not what you're rating. Marketing doesn't necessarily understand their games and what's interesting about it. And occasionally a game is fascinating despite what their creators were trying—Jim Rossignol loving the deeply buggy unpatched release of Boiling Point for its sheer constant surreality comes to mind as an extreme example of that.

N'Gai, it's far too early for me to do my You Don't Need To Complete A Game To Review It piece, I suspect. Methodology of reviews is a question all of itself.

***

Stephen Totilo, MTV News: I wonder why Shawn dragged me into this. I seldom write reviews. I don't put scores on games. My main gig's reporting, a.k.a journalism, a.k.a. the thing most people don't really mean when they want to talk about "games journalism" because the thing they really mean to muddle over and improve upon is what we're talking about here: games-reviewing. I'll give it a go, nonetheless! Scores, who are they for? What do they do?

The question we're answering is whether those who review games pick a number before writing a word. Kieron says the ideal reviewer would not; he and Leigh agree it's hard not to pick a figure already. Shawn's acknowledging the humanity of having preconceived notions but dodging his own question about whether that made him start with a number. But I guess it's hard in some ways to pick a figure at all when it's so unclear what the point of it is.

What does it mean to select -- prematurely or even at the "right" moment—a seven for a game? Or to see a game and, at first sight, have your gut gurgle that it’s a nine?

A review score number may be for the fans, a shopping guide metric that informs a purchase or justifies one already made. It may get used for the dastardly purpose of comparing a game to another—even though it never quite works to pit a 2008 sports game that got an eight against a 1998 role-playing game that got a nine, especially if neither is as good as Tetris. A numerical score might, in isolation, even indicate if a game's any good, but not always.

We're talking about arriving at a number, and, frankly, I don't know how you all do it. A decade ago I worked at a boxing magazine and sat in press row for many fights. Scoring vexed me then. I'd score rounds for my coverage on the "10-point must" system: 10 for the winner of the round, nine for the loser unless he got knocked down or really took a beating, which would dock him to an eight. In that system we see the Gillen-described method of scoring-by-reduction. We also saw the great gaming tradition of grade inflation. Give a judge (or a reporter aping the actions of the official judge) a 10-point scale and all kinds of psychology comes into play.

The other thing I saw at the fights—the thing that really stuck with me—was how hard it was to score any of it. Boxing matches aren't like Rocky fights. It's often hard to see who is winning or which fighter is doing the better work. Sometimes it's all boring or repetitious, but you still must score each three-minute round. Putting numbers on these things—and the official judges had to, in case it went the distance and, god forbid, the paying public needed to know who won—was a murky and unpleasant job. Try it some time. I'd root for the knockout, which would render scores moot and sweep any errors in numerical judgment away. The scorecards didn't matter then. Any scoring biases we had would be secret. The fallacy of putting a number on things would be dodged, and everyone would go home happy. No one would have to know that I gave a 10 to fighter B because I felt bad that he'd gotten beaten up for the three previous rounds or that I gave the wrong guy the first round because I bought into his pre-fight hype.

***

Robert Ashley, freelancer: I took a break from enthusiast press game reviews for a couple of years. What a fucking relief. No more death threats from insane superfans who think my evaluation of their favorite game is some kind of paid-for hit job by a shadowy corporate network. No more forcing myself to play through a 40-hour game in three days. No more tearing my hair out trying to avoid the clichéd language of a form of writing frozen in its awkward adolescence 15 years ago. Free to play whatever I wanted, I fell in love with games all over again. Hard.

Now that I'm back and picking up the occasional review, I simply refuse to engage in the bullshit that used to drive me insane. Review scores have one use: driving traffic from message boards and social networks to your site and giving those people an excuse to argue out their fan beefs in the comments section. I treat them as such.

I have no methodology for choosing a review score. I certainly don't think about it much. Your gut feeling (after either beating the game or the game beating you) is more accurate than whatever you might come up with after careful consideration. This is how the rest of the gaming community arrives at an opinion—and probably why so many people feel that critics are out of touch. When you sit at your computer, running down all the plusses and the minuses—technical issues, story concerns, lovable roughness, annoying roughness—you can end up talking yourself into a score that doesn't really represent your true reaction. You can't explain the magical pixie dust that made the empirically bad game good. You can't explain the soullessness and sterility that made the empirically good game bad. You let your stupid logical brain take the wheel and explain yourself into a lie.

When I say you, I mean me.

Anyway, I say be gutsy and honest with a score, and save your careful thinking for the text.

***

Jeff Gerstmann, Giant Bomb: Well, I won't deny that scores stir up message boards and social networks and such. But to claim that's the only reason they exist is a pretty narrow, jaded view. I think scores are primarily there to serve as shorthand for folks that won't or can't read the full review. They're meant to serve as part of the summary. A deck, a score, and, depending on your publication's review style, some pros and cons or whatever. They aren't rocket science, and were never really meant to be treated as such. The key is to not let the different ways that scores are misused get in the way of what you're trying to accomplish with your reviews. I don't care if the scores I give fit in with the rest of the industry on the review aggregator sites. I don't care if people infer the score to mean that I'm playing favorites because I'm obviously "TEH BIAS" or whatever. I care about the people out there who haven't been following a game from day one, and the people who haven't already pre-ordered the game and are just looking for validation. As soon as you start bending your review systems in order to cater to those extremist segments of the audience, you're getting away from the thing that reviews are designed to accomplish: assist average, everyday people in their purchasing decisions.

I say assist because we've reached a point where one review can't possibly work for every single person that reads it. The audience for video games is too widespread and varied now for reviewers to think that their review is the only one that matters, or that it will be able to directly state if a person should or shouldn't buy a game. This, more than anything, is what should be driving a change in the way games are reviewed, not a bunch of reviewers who are tired of all the weak-ass game review clichés that are still out there. Getting rid of scores because people who write reviews are tired of assigning them and dealing with the fanboy rage that invariably ensues hurts the consumers that actually use reviews for their intended purpose.

But to answer the core questions, I don't really think too much about scores when I'm playing a game. I attempt to go in feeling cautiously optimistic about the game in question, and as I'm playing, I think about text, and things in the game that need to be specifically called out. I start to think about the best way to mention those moments, and the best way to call out its flaws. At some point, all that text swirling around in my head starts to sound like a range of scores, so maybe around halfway through playing a game I start thinking a little more about the score. But it isn't until after the review is written that the score is actually assigned. The score is meant to sum up the text. If I've just written a review full of harsh criticisms, well, then that sounds like a pretty low score. Assigning a score and then attempting to justify it with text puts the cart before the horse.

Assuming a score (or range of scores) before actually playing the final game is pretty dangerous territory. Carefully controlled publisher-run demos usually paint a pretty rosy picture of a game, and games often don't live up to that. Case in point: every time I saw Mercenaries 2 prior to its release, I thought it looked awesome. The missions seemed smart, the co-op was fun, and it felt like a game that would offer a lot of variety. The final product turned out a collection of dopey missions that showcased the game's boneheaded AI, the co-op didn't make much sense, and a lot of the missions were pretty boring. I didn't review Mercs 2, but not letting pre-release exposure to a game color your review with overt disappointment or a sense of smug "I totally called it" satisfaction can get a bit tricky.

So I agree that, ideally, a reviewer should start with no preconceived notions about a game based on budget, hype, promises made by the developer, and so on. But at the end of the day, we're all human, and I'd expect that some form of disappointment over a game that fails to deliver on promises or excitement over a sequel that's turned out better than the last leaks into some of our reviews. The key is in owning up to that and presenting your reviews as informed opinions, rather than hiding behind the old paradigm of rigid objectivity.

***

Shawn Elliott, 2K Boston: I didn't mean to duck the question, Stephen, and I definitely don't start with a specific rating in mind. However, I'm sure that I have imagined ranges of scores that a given game would receive whether I or anyone else was to write the review. That's not to suggest that I once forced the square peg of a game to fit the round hole of my presumptions. I never did. Or I don't think I did. What I'm acknowledging is that, all the same, something was on my mind, both before I began and while I was playing. I think this is the case for every videogame critic. And while that something isn't necessarily decisive, it's nonetheless worth investigating.

I should also add that our predictions regarding meta-ratings and the reviews of other critics are on the mark more often than not. (In these instances, self-fulfilling prophecy isn't an issue.) Some companies are so confident of our ability to make these calls that they're willing to pay us for our input as consultants.

Jeff is correct in that sometimes PR-controlled preview demonstrations are smoke-and-mirrors magic shows. But what about when we're allowed to play near-complete code for prolonged periods? I'm not talking about performance issues—commenting on the framerate of an unfinished game is almost as pointless as it is for an Entertainment Weekly writer to assure her audience that King Kong may or may not appear in place of a green screen. Sometimes design, locked down years prior to a game's preview phase, is apparently dopey. Again, I have to emphasize that holding some assumptions in no way necessitates my maintaining them in the face of final evidence.

You also imply that an aversion to cliché shouldn't drive change in the way that we review games. I won't argue that cliché is the one and only reason to reconsider our habits, however, I count it among the many. The paragraphs on a game's graphics, sound, and so on in previews and reviews produce recognizably generic writing devoid of the discovery and perception that might make them worth reading. They are lazy in that they eliminate both the need to transition thoughts and to interpret a game as the complex product of interconnected components (instead of simply summarizing these parts).

Even worse is when the paragraphs that constitute a template are themselves composed of yet more methods of avoiding actual analysis. I mock the overuse of words such as compelling not because there is anything wrong with the words themselves but rather with the way that they're used to replace real explanation. We know that any guy in the game store can say he likes or doesn't like a game's graphics or story. We recognize that it's our responsibility as paid writers to say something more than "I like it" or "it's good." Replacing "like" and "good" with "compelling" isn't even trying.

***

John Davison, What they Play: If nothing else, review scores serve as the starting point of a discussion for readers. As Jeff says, they serve as a shorthand for those that have no interest in digging deeper than a fundamental thumbs up or thumbs down gauge of quality. I think we can all safely assume this, but back in my time at Ziff we experimented sufficiently that we got absolute, empirical proof.

Jeff Green and I spent a lot of time talking to Computer Gaming World readers, and trawling through our message boards to really try and put together the ultimate reviews section for the audience. We wanted to do something a bit different, but more than anything we wanted to acknowledge what a large group of our readers were telling us. That was, essentially, that "we're older" and "we're smarter" than the average gamer, so "treat us like that." They wanted longer, more considered think pieces about games, and it appeared, anecdotally at least, that review scores were not high on their list of priorities. They wanted, they said, to really understand what the reviewers were trying to convey. They wanted to really dig in.

So we gave them that. We took the scores off, and made the reviews longer. We actually went a step further, and tried to acknowledge the broader critical spectrum, and talk about what caused other reviews to express particularly positive or negative comments. It was our own little expression of idyllic critical idealism. A utopia of reviewing and we dreamt that it would spark enlightened and intelligent debate about specific qualities and opinion.

The reaction was spectacular. The readers really, really fucking HATED it. The most common complaint (I'm paraphrasing, but it was pretty consistent) was "How do I know what you think if you don't give it a score?" That and "you guys are retarded." We figured at first that it was simply a bit of culture shock and that it would wear off, but the negativity increased over time. After three months or so, we had to go back to putting a score out of five on the reviews just to stem the tide of vitriolic hatred.

On a separate note, I was speaking to someone recently who had some connection to Rolling Stone, and he told me that the reviewing process for albums there was that the critics only submit the text, but do not submit a score. The number of stars is assigned by the reviews editor based on the tone of the review. He was drunk at the time, so might have been talking out of his arse though. Does anyone know for sure if this is the case? Even if it's not true, it's certainly an interesting approach—and something I'd like to discuss in this context. If a reviewer is freed from thinking about assigning a score, but knows one will be applied later—would it necessitate a more disciplined approach to how thoughts are expressed? I know it would for me. But are we ready to relinquish that kind of control?

***

Robert Ashley, freelancer: I don't advocate putting an end to scoring. Scores seem to be the one thing that today's online audience can easily form a conversation around, and I think, ideally, a review should be like a conversation between reader and critic. I just think the incredible seriousness surrounding scores (born in no small part, I'm guessing, from the fact that retailers stock their shelves based on review scores, ratcheting up pressure on critics to treat scores like jury sentences) is irritating. Handing out a mediocre score to a mega-hyped game can brand you a heretic (a Crispin Boyer, if you will) when you're just trying to be honest about your reaction.

***

Stephen Totilo, MTV News Multiplayer: Who is actually upset about review scores?

Offended publishers who wanted a 9? Stressed developers whose bonuses depend or ability to get another deal require them to get at least an 8? Superfans whose feelings you hurt by giving the game they are going to buy anyway a 7?

Before we do any more debating about the merits of putting a number on a score I want to know who cares.

Have any of you come across gamers who won't buy a game they were curious about because you gave it a 7 and not a 9? And, if so, would they have made a different decision if your review didn't include a score? Was it review scores that did in Too Human or put Wii Music slow out the gate?

Do scores ever really hurt or help games? Or are we just debating the best way to describe a game's quality, be it through numbers, words or faces in various stages of excitement?

***

Leigh Alexander, Gamasutra/Sexy Videogameland/Variety: Backing Stephen on this. Scores do stress the reviewer, but the stress doesn't come from any source that really counts—excepting maybe the anxiety I get sometimes knowing people's jobs depend on, say, Metacritic, and hoping that I was as thorough and fair as I could possibly be.

Like some others who've chimed in, I write the text first, and then see what score the text supports. In a way, I'm not assigning the number as a be-all measure of the game itself, but as a shorthand for my evaluation, and I think that's the function for which scores are most useful—as Jeff said, a single value that makes sense from a macro viewpoint for people who don't read text. There are a lot of those people (which makes me wonder if we shouldn't be trying to create more accessible, readable text, but that's probably a whole 'nother issue).

Finally, because I talked about preconceptions, I just wanted to clarify that while I may sometimes go in with ideas, hopes, dare I say biases, I never go in with a number in mind. The number is the last thing I come up with.

***

Dan "Shoe" Hsu, Sore Thumbs: To Stephen: Yes, yes and yes. They all care. But in the end, that review and score aren’t for them anyways -- they’re for your “normal” audience—so it really shouldn’t matter how bunched up their panties get.

At EGM, we’ve had plenty of readers who told us they would not even consider a game purchase if the reviews didn’t average a certain score they had set in their minds. Now, this “certain score” is usually a moving target—higher if it’s a game that reader wasn’t originally interested in, lower if he already had that title on his Amazon wishlist. This overreliance and faith in this one rating (scored by someone you probably don’t know intimately well) may seem silly, but I’m absolutely with Jeff and John on this: Despite complaints from a vocal minority, the vast majority of readers really want that number, letter, or direction the thumb’s pointing. It’s ingrained in society and it’s pointless and stubborn to fight it. People don’t always have time to read a 2000-word, well-crafted review to get inside the brain of the reviewer. For most folks in this short-attention-span world, that “4 out of 10” usually says more than enough.

Are we off-topic here, by the way? To answer question #2 above, yes, I sometimes change my score after I write the text. I do it like Robert initially: I score with my gut. But then while writing a review, I get to reflect upon my play time, think back to my progress 10, 20, or 30 hours ago, check back on my notes from last week, etc....and then I might adjust my score (usually by one increment up or down) based on my experience in its entirety. This may seem obvious—to score the whole game, not just how it finishes -- but I also have that same short-attention-span problem. And this helps me keep too much emotion out of the process, too, so a game’s high-note finish doesn’t unduly inflate the score.

***

Francesca Reyes, OXM: I can’t tell you how much I personally hate coming up with scores. It’s putting a quantitative label on something that’s qualitative. But that said, it’s a necessary and completely understandable function of game reviews in enthusiast pubs. (Hell, when I read movie reviews or book reviews, I do the same thing—look at the score to justify whatever half-ass, cobbled together pre-opinion I may or may not publicly admit to having.) The reviews I’m used to writing aren’t criticism in the sense that you’re a “reader” and you’re “reading” a game like you would text, or a movement, or even a movie—they’re practical forms of consumer advice. So, I agree with Jeff in that we’re here to inform our audience if a game is worth X amount of money based on whether it delivers, and to what extent, on its back-of-the-box promises, as well as how it handles as a game. And that demands a score, really. A signpost for what the review contains.

I’ve been lucky that all the pubs I’ve worked for fall in the realm of “enthusiast,” so in a lot of ways, you’re writing for gamers like yourself when you score a product. You’re also using a set of established criteria, depending on your publication’s ranking scale. If a 7 in a certain magazine means “average,” while a 5/5 stars in another means “must-have,” etc., this is what you’re working off of.

I remember writing a review of one game across multiple publications. One had a 100-point scale, another had a four-star scale, and yet another had something else. The thing that sticks out in my mind is that the 100 point one was for Ultra Gameplayers and the star-scale was Next Generation. UGP was your regular “review as a gamer.” Next Gen was “review for innovation and uniqueness, like ‘does this push the boundaries of its genre, etc.?’”. One game, reviewed against two very different sets of criteria. Interesting contrast.

So yeah, it’s totally ideal and utopian to think that you can sum up a game in a one-word or one-sentence definition from a pre-existing list of rankings, but this ain’t science. It’s voodoo magic in a lot of ways, no matter how hard we try to justify what the numbers, letters, or stars mean.

I try not to go into a review with a preconceived score in my head, but like Shawn said early in the thread—the result of months of pre-release hype or non-hype may or may not play a role in expectations from the reviewer and the reader. You may never really know. It’s just human nature, really, and publishers know this. But the trap is that we all play a shit ton of games, right? But our readers do not. Yeah, some of them play a lot of games that we haven’t. Some of them may play as many as or more than we have. But that’s the small portion of our audiences. Most of them pick and choose what they buy and we have to understand that spending their cash may rely heavily on what reviewers say. You have to respect that and go in to a game with the same expectations that someone without the months of exposure to a title might have.

Maybe that’s idealistic to expect this of writers who are supposed to be “experts” on their field (how do you become an expert in a field or medium if you’re not exposed to everything it has to offer, right?), but when it comes to reviews—in a lot of ways the boss is your reader and you have to kind of get in their skin. Am I always successful at doing this? Hell no. But it’s what I always aim for.

As for the actual process of coming up with a score—sometimes you just know, based on games you played before or a gut feeling when you’re playing it. I sometimes wait to put the score in the review until after I’ve written it so I can step back and get some perspective. If I have the luxury of time (shyeah), then I can let it sit for a bit and return to it for another pass to see if the score still holds. I like how Shoe mentioned the “high-note finish” and I agree. Games are experiences and once you see the entire narrative a developer has to tell you, there’s a sense of accomplishment that sometimes make you review through rose-colored glasses.

***

N'Gai Croal, Level Up/Newsweek: Robert, I’ll raise my hand and say that I do advocate putting an end to scoring. You wrote, "Scores seem to be the one thing that today's online audience can easily form a conversation around, and I think, ideally, a review should be like a conversation between reader and critic." Yes, a review should be a conversation between reader and writer. But /what kind/ of conversation do review scores foster? Judging by the Metal Gear Solid 4 example Kieron cited above, not much. (We’ll get into that more when we tackle Reader Backlash.)

Jeff, you wrote, "scores are primarily there to serve as shorthand for folks that won't or can't read the full review." Unless I'm missing something, that's quite an indictment of a portion of your audience. Why would you write for people who won't or can't read an entire review? You also say that scores "assist average, everyday people in their purchasing decisions." I don’t object to heds and deks; pros and cons; bullet points; final words; buy, try, fry; and other forms of giving readers a succinct take on a reviewer’s opinion about a game’s value. Heck, you could just tell them how much you think the game is worth. And like you, Fran, I firmly support the consumer guide function of game reviews. But anyone who won't or can't read an entire review isn't making an informed decision by looking at a single review score, so I’m not convinced that the reason review scores are “necessary” is to genuinely inform those consumers who can only focus on a letter grade, a number or a star rating.

In a different context above, Shawn talked about intentional fallacy. I think that's what's happening with review scores in the age of the Internet. Those of you who assign scores intend them to perform a certain function, but in the real world, the use to which they're being put by the most vocal portion of your readership is pernicious. Scores help bring out the worst in readers. They shut down conversations; foster silly debates; and they encourage meaningless comparisons. For too many readers, the very presence of scores turns the text of a review into a sideshow for the main event: this number, those stars, the orientation of that thumb. The text becomes a caption and the score becomes the photograph, en route to becoming the final cog in the Gamer Metrics-Metacritic-GameStop machine. Why would any of us want to perpetuate that?

This may seem easy for me to say from the perch of a blog and a magazine that isn’t dependent on gamers for its survival. But many of us work for or were previously employed by outlets that have been struggling. None of us are safe. Newsweek had a round of buyouts this year, the third in my nearly 14-year tenure at the magazine. Time Inc recently had layoffs. MTV had layoffs last week. Ziff-Davis closed Games For Windows magazine. The Tribune Company just declared bankruptcy. For years, magazines and newspapers have been moving towards bigger photos, more charts and even “charticles.” The Associated Press has imposed a 500-word limit on its entertainment writers. All of this devalues the importance of the word. Review scores are yet another signal to your readers that your words don’t matter.

If we don’t think scores are genuinely meaningful—there may be a robust defense of the inherent value of review scores; of the 6.5 versus the 7.0 and the 82 as compared to the 89, but no one has offered it yet—why do we continue this charade? Shoe, you wrote that the desire for scores is “ingrained in society and it’s pointless and stubborn to fight it” and that “For most folks in this short-attention-span world, that ‘4 out of 10’ usually says more than enough.” Apparently it’s not enough for most folks that we slit our own throats; we’re expected to provide the knives as well. I won’t pretend that yanking review scores will bring an age of genteel conversation or Socratic debate to the intertubes. But if message boards must be clogged with pointless argument, I’d rather it be fuelled by the words you wrote rather than the numbers you assigned.

***

Kieron Gillen, Rock, Paper, Shotgun: Since you ask for a justification for scores and the fetishized graduations of hundred-points scale, N'Gai, let me give it try. I'm from the wanky English tradition, and was anti-scores for most of my adolescence for the obvious reasons. They're stupid and putting any kind of scale over a subjective experience was ridiculous. For game reviews, at the current point in the medium, I've come around to them on solely utilitarian grounds.

For scores, per se: reviews exist fundamentally as buying guides, the proverbial shit-filter. They can do other things, but the reason to exist is to spend other people's money. When I score, for the pure consumer magazine (as opposed to one with a more critical leaning) I score for one thing only: should you buy it. For those less-prominent games, a high score is a direct tool to even make them read it - World of Goo and Braid getting a string of 10s is an obvious flare for ATTENTION! in a way which just giving them a glowing review isn't. If reviews are a shit filter, the mark is a very blunt tool for achieving that. That, to many gamers, is just what they want. In a real way, N'Gai, that's the job.

(This will come up later, but the thing about classical game reviews is that it isn't like film or music reviewing, but a hybrid between some stuff that's purely subjective and some stuff that's objective. You don't get a band's new CD which doesn't work on most CD players. You don't get movies which freeze randomly. For a review, that stuff genuinely matters and we're betraying our readers and being deeply disingenuous if we pretend otherwise. Are there other forms of games writing for other readers? Hell, yeah. But dismantling the review isn't the solution. We should just go and build something else.)

And the hundred-point scale? Perversely, what I most like about it is actually its weakness. It's inherently ludicrous. Who can tell the difference between 83 and 86 percent? No-one. In other words, its subjectivity is totally clear. The 10 or 5 point scale has a way of actually tricking people into thinking there's some science at work. The hundred-point scale--and calling it "percentage" scale is another thing that's deeply deceptive—is very silly. The fact it's a rough tool rather than a scientific implement is blatant. If you try to argue a few percentage scores you look as if you're suffering from some obsessive-compulsive disorder in a way you don't necessarily look like if you try to argue between a 7 and a 10. 100-point scales, treated correctly, are an ideal way to both act as a shorthand for the review, and simultaneously make it clear the mark isn't the review.

***

Shawn Elliott, 2K Boston: To add to N'Gai's thoughts....

Some of us suggest that our audiences sees scores as buyers' advice. Actual sales rarely correlate with review scores in cases where games are not also heavily hyped and marketed. Increasingly, gamers pre-order games prior to the publication of reviews. Interactive demos allow our audiences to decide for themselves whether or not a game will be worth their dollars. In addition, word of mouth and message board discussions inform our potential audiences' purchasing decisions with an intimacy and directness that we cannot provide. Finally, review aggregation sites such as Metacritic mute the bias of individual reviewers and provide a bigger picture.

I suspect these circumstances suggest that our self-perception is, well—a throwback to a time when magazines and websites were gaming's gatekeepers.

And yet we have John's anecdote about the angry reactions of some Computer Gaming World readers when the magazine dropped its scores. Robert introduced the idea that "review scores have one use: driving traffic from message boards and social networks to your site and giving those people an excuse to argue out their fan beefs in the comments section." Jeff countered that ratings "are primarily there to serve as shorthand for folks that won't or can't read the full review," which prompted N'Gai to ask why anyone would want to write for such an audience. Maybe our audiences aren't a homogeneous monolith—not in the sense that different readers look to a Level Up or Gamespot for different reasons, but that we (and our bottomlines) want or need different readers to look to a single site or magazine for many reasons. Is this part of the problem?

***

Stephen Totilo, MTV News: Shawn concluded his note with "Is this part of the problem?" I ask again, as I did in my previous note (with slightly different phrasing), what's the problem? Who or what are review scores hurting? N'Gai makes a passionate argument against the damage he sees review scores doing to the discourse about games on some message boards and comment threads. And he pitches a convincing case that such damage obscures the value of the scorers' words. All told, though, that doesn't seem like a whole lot of pain.

But, again, what's the problem? Are quality games not being appreciated because of the existence of review scores? Are quality critics not being read because of the proliferation of scores? Are talented game creators losing their jobs because of review scores? If yes to any of those questions, then would the abolition of scores remedy those situations? If not, I see no more reason for Giant Bomb and IGN to ditch scores than I see them needing to have their reviewers append to their reviews drawings of whatever flower the game they just played makes them think of. Whatever info the readers find useful and edifying, you know?

To the score haters, though, I direct you to Kotaku's reviews for support to your arguments. The mad bloggers there found a way to write reviews that don't use scores but can still somehow be comprehended in the time it takes to tie one's shoes. See their Far Cry 2 review, and, aside from too many puns, it gets the job done: http://kotaku.com/5071946/far-cry-2-review-hurry-boy-its-waiting-there-for-you

***

John Davison, What They Play: Kieron's point that game reviews have to be a hybrid between purely subjective commentary and some stuff that's objective is an extremely important one. The innovation and creativity of games design (or lack thereof) is more than just artistic expression, and an assessment of the overall experience is often incomplete without some mention of the mechanics. Criticism of games reviews often focuses on the fact that we spend too much time on this stuff, but it is often a more important consideration than the "art". I think we'd all love to spend more time on digging into what a game is trying to "say"—but we're often faced with impenetrable control mechanisms, or distracting technological problems that cannot go unmentioned. There are also still an awful lot of games, particularly in the current climate of "casual" style games on DS and Wii, that are practically *all* mechanics, and an intelligent discussion of how well they work is all that's needed.

Take a game like Prince of Persia, though. While there's much to be said about the art style, the way the narrative unfolds, and the co-dependence of the lead characters—a review of the "experience" cannot ignore the fact that the game has some nuts and bolts mechanical issues. It has a tendency to play itself, for a start. The controls are simplified to the point that it can be distracting to an experienced player, and that there are some unavoidable issues with the camera. They're boring topics compared to something that, say, focuses on an emotional response to the player's growing bond with Elika, but I think there's an expectation from our audiences for us to convey and assess the overall experience, not just the "heart."

Shawn, your point about the "homogeneous monolith" is a really important one. Times *have* changed since the days of magazines and websites being gaming's gatekeepers, and while we all tend to identify with our own outlets, and maintaining a sense of community that is distinct, we all have a part to play in the broader scheme of things. Given that Google is the window through which the world views the vast majority of content, there's less and less loyalty to specific outlets, and instead people are simply looking for opinions on specific topics. For this growing audience, a score is still important for exactly the reasons that Robert and Jeff have mentioned. It's shorthand. It's a hook. It's a way to get people to look at our reviews. We are all, after all, businesses. We need traffic. We need people to come to our sites and read our stuff.

N'Gai, you asked "Why would you write for people who won't or can't read an entire review?" I think that's oversimplifying. Often, our audiences (both the hardcore, and the not-so-hardcore) go trawling for review scores purely for validation of their taste. An abundance of high scores serves that purpose, and makes them feel vindicated whether it's because they've made a pre-order, or are simply fans of the franchise.

***

Robert Ashley, freelancer: I wouldn't want to separate "art" from game mechanics and interactive feel. Atmosphere and storytelling have been a fixation for games critics lately, but the connection between the player and the machine—the mysteriously engrossing feeling that makes people scrunch their faces up and open their mouths like zombies—is the real "art" of gaming. Technology and all its various failures play a huge role in that feeling. Anyone can tell when something isn't running smoothly, but I wouldn't call these observations entirely objective. Badly behaving cameras can drive some people insane. Other people not so much. The same thing applies to many common technical problems. The question is, did you have that great zen feeling of being inside and part of the game, or did technical problems and poor interface constantly break the spell? Making a game feel like Super Mario Galaxy isn't a simple question of technology and time. There's something special going on there.

***

Tom Chick, freelancer: On the subject of review scores and expectations, I have a lot on my mind when I review a game. I'm crammed full of preconceptions, expectations, prejudices, hopes, and fears. I call it "context" and it's probably the single most valuable thing I have to offer as a guy who writes about games. I've got thirty years of it under my belt. I don't let loose of it when I play a game, and I certainly don't let loose of it when I write about a game. I write as a hobbyist, and I write mostly for other hobbyists. We know how Spore was hyped, we're aware that Haze was from the Timesplitters guys, we've heard about the problems with Killzone, and we remember Trespasser. It's how we talk. When I have my druthers (i.e. when I'm not writing for a wider audience), it's exactly the sort of stuff that goes into what I write. And, yeah, it figures into whatever number or letter I have to slap onto a review. Like any gamer worth his salt, I have the bias of experience.

As for how much and when I think about scores, I think about them all the time. I think about how much I hate them and how much damage they do to the state of videogame discourse. Scores are an end run around saying anything meaningful. I hate when someone says (almost always on the internet), "I liked your review, but I would have given it an 8 instead of a 7". Because that's an unborn conversation that will never happen. If I didn't have to come up with that insufferable 7, the comment would have had to go as follows: "I liked your review, but I disagree with what you said about it being too hard" or "I liked your review, but I disagree with what you said about the graphics being too much like Fable" or "I liked your review but I disagree with what you said about the ending feeling out of place" or even "I liked your review, but I liked the game more than you did". Those are all starters for at least a line of thought and at best a conversation, and in either instance, we can both be the wiser for it.

Review scores are for the lazy, the unengaged, and the inarticulate. They're for stickers on boxes and press releases. They're understood differently by different people, and they're applied differently by different publications. They're an attempt to inject some sort of science into someplace it doesn't belong and the sad irony is that they mean nothing. I don't know if games are art, but so long as we're branding numbers into their flanks, they're certainly consumer products.

Now I have the luxury of saying all this, because unlike some of you (well, John and Dan, at one point), I don't have to run a magazine or website. To folks dealing with lazy, unengaged, and inarticulate readers, I don't envy you your job selling stuff to them.

***

N'Gai Croal, Level Up/Newsweek: Harry, you came up as a journalist and critic alongside hip-hop. You wrote for The Source, with its famous one-to-five-mics rating system and all the arguments—inside the magazine and among its readers--that were stemmed from its reviews. You also worked for the Village Voice, an outlet that didn't apply scores to any of its reviews—with the notable exception of Robert Christgau's monthly Consumer Guide, which introduced letter grades to music criticism—yet fueled legendary debates nonetheless. You worked for Rockstar Games. You're even writing a book about architecture in computer and videogames. (I bow.)

What do you make of all of this? There's nothing new under the sun, so you must have been in or around discussions like the one we're having right here—does it bring back any memories you can share? Are Jeff and Francesca correct when they say that scores are a legitimate part of consumer reviews, regardless of how some readers may respond to them? Are Tom and I right to argue that review scores help engender the all-too-often juvenile discourse that surrounds videogames? Is Stephen onto something when he suggests that the case against scores is tenuous, and that as long as some readers find scores edifying, reviewers who choose to score games shouldn't agonize terribly over doing so? Is John right that Rolling Stone's editors, not its writers, assign the star ratings? (I had to try, man.) And finally, what's it like being on the receiving end, watching scores and reviews trickle in, and—tell the truth, now--which mattered more?

***

Shawn Elliott, 2K Boston: Before we finish this section with a final question on the real consequences that review scores can carry (an elliptical response to Stephen's repeated "What's the problem?"), I want to take a moment to address any unanswered inquiries. Note that, for the moment, I'm withholding my thoughts on the intersection of mechanics, meaning, and the stories that games try to tell (the short, unsubstantiated version is that I agree with Robert that it's largely fruitless to look at these in isolation).

Kieron commented that he thinks "AAA-popular-sequels tend to start with 9/10 and lose marks, while games with less expectations start with 5/10 and have to gain them." This strikes me as especially true for enthusiasts. Anecdotally, it also seems as though critics are more inclined to take the gloves off with less-anticipated, lightly marketed games or, conversely, to forgive their faults. In addition, Leigh wondered about the wisdom in bringing an awareness of a game's budget to our analyses. Thoughts?

And then there's John's Rolling Stone rumor. What do we make of a critic submitting copy and his or her editor supplying the score? It's an interesting thought experiment. I do think that the policy would encourage writers to explain themselves more carefully, and that it might erase much of the discrepancy in words and numbers, but what ramifications would it carry?

***

Jeff Gerstmann, Giant Bomb: I find Kieron's comment about big budget games catching a bit of a break in reviews to be possible in isolated cases, but I don't think it's the norm at most publications. I've also seen the opposite, where the big budget game gets trashed for not living up to the insurmountable mountain of marketing hype while the low-budget indie darling catches a break because it was made by a team of five people or something. And I've certainly seen little, unmarketed games get absolutely thrashed in reviews. Sometimes it seems like the reviewer is doing this because it's "safe" to do so, like it was part of some sort of "see? We totally use our entire 1-10 scale" chest pounding. This is why it's important for reviewers to have an editor (or editors) that can keep them in check and ask questions about a review and its score before it gets published, especially at outlets where that one review is meant to represent the entire publication's view on a game.

While I've never been a party to anything quite like this Rolling Stone rumor, I've spent a great deal of time as an assigning editor for reviews. The most important part of that position is working with the authors on clarity, to make sure they actually mean what they say, and that they aren't coming off as more positive or negative than they intend. Sometimes that involves changing the score to make it match the words more closely. This would happen most often with freelancers, as they can't really be expected to be experts of how one publication's scoring system differs from another. But except in extremely rare cases where the reviewer wasn't available at post time, those changes were made after discussing it both with the internal staff (as part of a review vetting process) and with the author. I wouldn't want to do things the way Rolling Stone supposedly does them, but that might say more about my faith in most freelancers than it does about the policy itself.

Lastly, I really don't think a game's budget matters when reviewing a game. High-budget console games and mid-budget console games cost the same $60. The only dollar amount that matters is the retail price. While bad games certainly don't get dramatically better as their retail price drops, it's a lot easier to overlook some of a game's flaws if you're getting it for $20 instead of $60. But at the same time, it's probably fine to mention a game's budget in passing. There's a big difference between a brief mention of the budget and using it as the centerpiece of your entire article.

***

Leigh Alexander, Gamasutra/Sexy Videogameland/Variety: I'm super eager for Shawn's discussion point on the impact of the scores, and I don't want to drag everyone down a side trail, but wanted to note something on the Rolling Stone rumor.

No idea what they do at Rolling Stone, but I've actually been in situations more than once where the final star ranking or number was suggested by my editor. It's not as sinister as one might imagine—it isn't as if a person who didn't play the game is independently applying a number with no input from me. In fact, it's more of a collaborative review process between the editor and the writer to be sure that the final rating really does correlate to the text as it's written. Limits in what you can express within a word count, which I'm sure Rolling Stone is constrained by, can make it useful to have two pairs of eyes on the situation.

In fact—and I think this is especially true for those who write for more mainstream print—many reviewers in that context want to avoid the wham-boom core market number hysteria. They want to write articles, they prefer to write crit, and don't want to calculate numerics. They may be non-traditional reviewers (as I'd assume Rolling Stone's are). In that case, especially where Metacritic is involved, in my experience an editor may volunteer to apply a score that's correlative with the review text simply to offer an option for those writers who don't want anything to do with the numbers game.

In those cases I have always elected to have input because I feel comfortable that way—but keep in mind that scores can cause headaches and PR arguments for reviewers, and in that case, an editor stepping up and saying "point them to me if they hate the number" may be one way of allowing the writer to do his or her own most honest work with impunity. When an editor plays a role in the score, he or she's essentially "backing" the writer's text by shouldering all those burdens a number can provoke. It's a pleasant reversal from the nightmare scenarios we hear about where writers catch flack from their bosses or even get canned because some publisher flipped their lid.

Again, I don't even know anyone who writes for Rolling Stone. I wish I did, because I'd tell them to write about good music again. But my suspicion is that scenario doesn't so much involve some sinister overlord stealing scoring power from the reviewer, rather an editor who wants to make room for the writer to do what the writer's being paid to do—write well and thoroughly on a title.

***

Shawn Elliott, 2K Boston: The editor in me empathizes with Jeff, especially in cases where a green writer's copy is in apparent conflict with the rating he or she attaches to it. Discussion is helpful here, as the writer may reveal anecdotes and a level of analysis lacking in their initial draft. How many of us remember the teachers who told us they wanted to read in our revisions whatever it was we'd just said?

The writer in me, however, is just as wary of knee-jerk editing, and editors who've played -- and have formed their own opinions of -- the games they've assigned to freelancers. Here http://www.gamesradar.com/pc/call-of-duty-4-modern-warfare/review/call-of-duty-4-modern-warfare/a-2007110512090683184815/g-20070214121215902085 , for example, we find text apparently taken from PC Gamer UK's Call of Duty 4 review ( http://www.computerandvideogames.com/article.php?id=175050&site=pcg) transported to GamesRadar's 360 review page where a presumably miserly British 8.5 becomes a prodigal American 10. Bylines differ, too, as though we're to believe that two writers, an ocean apart, have arrived at the same sequence of words and altogether different assessments of their meaning (the editor's note states that the GamesRadar score was once a 9, which still doesn't explain discrepancy). That Francis's copy came first is fairly obvious in lines like, "If the whole game had been like that, or even just as inventive throughout, you'd find a frankly silly score at the end of this review. Instead it's a more restrained one[....]" My suspicion is that a comatose editor couldn't be bothered to read let alone edit the original. Frankly, that's a fucking insult to writers and readers if there ever was one. It says, 'Hey, fill some space for the cretins naive enough to not notice. We'll handle the hard work."

Hopefully, this is an anomaly. The idea of dismissing the Rolling Stone idea for it is as dumb as it would be to eliminate governorship because of Blagojevich, but, again, we're all human.

***

N'Gai Croal, Level Up/Newsweek: Regarding the Rolling Stone rumor, I'm think it's legitimate for reviewers and editors to discuss scores before assigning them, but I'm wary of editors assigning scores all by their lonesome. This point was driven home for me when, in the wake of Jeff's firing from GameSpot following his Kane & Lynch review (http://blog.newsweek.com/blogs/levelup/archive/2007/12/05/reflections-on-videogame-publisher-and-employer-contempt-towards-the-enthusiast-press.aspx), Tor Thorson issued a statement explaining that "The copy was adjusted several days following its publication so that it better meshed with its score, which remained unchanged." (http://www.joystiq.com/2007/12/05/gamespot-addresses-gerstmann-gate-concerns-in-depth/) Even given the force majeure of this situation, the question that immediately popped into my head was this: if changes had to be made at all, why wasn't the score changed to better mesh with the copy? If there are going to be scores, the individual reviewer should always be intimately involved in that process.

*****

Shawn Elliott, 2K Boston: To finish this section and finally return to Stephen's point that scores, despite 'damaging the discourse about games' and 'obscuring the value of words,' aren't an actual problem, I'd like to ask one last question.

Review writing carries real consequence because some publishers do base developer and PR bonus pay on aggregated ratings. This shouldn't concern critics, but once-warm PR people and game producers can become cold upon our publication of undesirable review scores, diminishing or eliminating our ability to secure subsequent interviews and access. Postmortem discussions and exclusive looks at the publisher and/or developer's forthcoming products are less likely. Conversely, a few publishers will permit us to post reviews before competitors, provided our review scores are favorable. Do such pressures produce a subliminal background—especially among members of the enthusiast press—or even enter our thoughts as we write or edit reviews and assign reviewers or scores? The stock answer says, "Only if you're a bad apple, and I'm not." But isn't the seeming impropriety of business in a bad barrel a problem in itself?

Consider special instances such as the Gamespot Kane and Lynch review that N'Gai cites. Jeff, are you contractually able to discuss that episode in any detail? John, Francesca, and Dan, as well, serve/served as EICs of enthusiast publications and presumably face/faced such pressures and repercussions.

***

Dan Hsu, Sore Thumbs Blog: Exclusive reviews stink...it’s a lose-lose situation. If you score high, then readers call shenanigans...even if there were none. If you score low, then the game makers get mad and won’t want to work with you in the future.

Plus, most companies want a guaranteed score or range of scores. We did play with this fire once at EGM...the very first time I was faced with it. I thought I could deal with it by protecting the reviewers from that discussion—I’d let them review the game independently and then see if the scores were high enough for us to secure that exclusive. But because I had final say on all magazine content, I’m still a part of the reviews process. In the end, I didn’t feel comfortable promising any certain scores, so I backed off and made a rule to never entertain these offers again. If we don’t even have those discussions to begin with, then we’ll have preserved the integrity of the reviews from start to finish.

Besides...if the reviewer caught wind of the deal, how can it not spoil things, however so slightly? Let’s say the reviewer’s 50-50 wavering between an 8.5 and a 9.0, but way back in the deep recesses of his mind, he knows that a 9.0 will get his website or magazine the exclusive review, which translates into revenue-producing traffic or sales. Will that load the dice? Maybe not for everyone, but that’s not the point. Once that process is tainted, it’s tainted. Just ask the reader what he or she thinks.

***

Leigh Alexander, Gamasutra/Sexy Videogameland/Variety: I think in Jeff Gerstmann's case, the fact that the editors took liberty with the final text and scores are evidence of the "Rolling Stone System" not working in practice—again, I support the idea of editors protecting their writers (as Shoe tried to do with his folks, it seems), and doing whatever they can do to keep the heat of publisher response to scores out of the way so that the writer can do his best work. The "protection" obviously didn't take place with Jeff, and while I've no inside knowledge of that situation it sure looks to me like they let him take a bullet, which is at the very least evidence that making any kind of bargain with a publisher regarding a review—its exclusivity or otherwise—is a BS proposition.

I hope that this reason, coupled with the evolution of Internet media, means that exclusive reviews are going to go the way of the dinosaur (or of the console exclusive, har har). I wholly believe we now work in an environment where quality and depth can be the primary competitive advantage, and less timeliness.

I also can't speak for Totilo, and don't mean to put words in his mouth, but when I agreed heartily with his response, I was agreeing with the idea that scores only cause problems for the industry and for us—neither of whom is the audience we serve, and therefore that they create stress is not a compelling argument against the fact that plenty of readers find them useful. Hell, I hate scoring as much as most of us do, and I prefer scoreless criticism and blah blah blah—but in all honesty, when I really want to know whether I should pay attention to a game, the very first thing I do is go to GameSpot and see what number they gave it, for whatever it's worth.

I never buy based on a number (let's take bets on how many—or how few—year-end top ten lists the 10-rated MGS4 or GTA IV get), and I don't think readers do either, which further takes the piss out of this idea of their damages. Believe me, I'd love to stop scoring. I'd love if we could enforce an industry-wide moratorium on scores, so that we wouldn't have to think about 'em and so that our audience could re-learn to focus on the words we write.

But I think we hate scores because of the undue importance that has been placed on them and the ways they've distracted from our work, and this is credited in part to editors balancing the needs of the publishers with their management of irrationally hostile reader reactions, neither of which should be their focus. None of this means they in and of themselves cause harm to the readership or to the art of the review.

***

Shawn Elliott, 2K Boston: Dan, do you see where this can apply to cases less extreme than exclusive reviews? Let's say that a publisher puts one game in the market this month and is also looking for an outlet to announce a forthcoming game a few months from now. No conspiratorial conversations or conniving occurs. Nonetheless, you have to know that by giving the available game a negative review, you could risk reducing the likelihood that the company will allow you to reveal their other title. Scruples aside, the cost-benefit equation is simple: A first look at the unannouced game is probably better for business than the bad review, provided nobody calls bullshit. And compromise comes in shades. An EIC can always assign the review to a specific staff writer who he suspects will appreciate the game more than his peers, and argue afterwards that he hadn't touched the score itself. I have no idea whether or not this happens. I do, however, know that commercial publishing is cutthroat.

***

N’Gai Croal, Level Up/Newsweek: From the outside looking in, I’ve been conditioned to expect that review cover = exclusive review = 8 or above. That led to a bizarre reading experience last holiday when I picked up a copy of Official Xbox Magazine with Turok on the cover, touting the first review. I can’t lie, I immediately flipped to the end of the review to see what score had been handed out…only to be shocked to see that it had been tagged with a 7! What went through my mind next was:

a) Damn, that took guts for Francesca to stick to her guns and let reviewer Paul Curthoys call it like he saw it.
b) there goes Christmas for Josh Holmes and the rest of the guys over at Propaganda Games, to say nothing of Graham Hopper and the folks at Disney Interactive.
c) I would not want to be the publicist who brokered that exclusive. Her/his job is to suss out the most favorable outlet when determining who’s going to get that first review, because you only get one chance to make a first impression. But a first review score of 7 means that something went terribly wrong—and from PR and marketing’s perspective, the “something went terribly wrong” in this kind of situation is /not /the finished game’s quality.

At last glance, the Metacritic rating for the Xbox 360 version of Turok was 69, so OXM’s review was right in line with the critical consensus as calculated by that aggregation engine. I’m sure that was little consolation for Propaganda and Disney, though.

John, didn't you write about the pressure surrounding review exclusives when you first started blogging?

***

Dan “Shoe” Hsu, Sore Thumbs: Shawn, that’s why this business is so messed up! Publishers want good reviews. Editors want exclusives. Magazines and websites want advertising. Advertisers want good reviews.

Sometimes, the companies know their products aren’t that great. They have internal research (via their own gut checks or “mock reviews”—early evaluations done by outside consultants) that give them a rough idea of how a game may score. If you’re dealing with a reasonable PR team and you’ve scored within their expected range, then you’re usually spared the grief. But woe is the outlet who’s at the bottom of the Game Rankings or Metacritic lists. We’ve had PR people complain to us that we’re “outside the average” (on the low end) on those sites, which John and I would always laugh at. Of course someone has to be outside of the average. If everyone was ON the average, there’d be no point in averaging them!

***

Jeff Gerstmann, Giant Bomb: Exclusives are yet another by-product of this business' print roots. Between magazines stories getting scanned, website text getting pasted into message boards, and the way that most of a site's traffic doesn't come in through the front door (where the big promotion of the exclusive always is), exclusive stories are waaaaay more trouble than they're worth. Exclusive reviews doubly so. On top of that, in my experience, most big exclusives don't really move the needle that much, traffic-wise. They've always felt like a huge waste of time to me. Attracting users by differentiating your coverage from the pack has always seemed like a better goal.

But are we talking about a problem that doesn't really exist anymore? Isn't IGN the only publication still doing exclusive reviews? And if so, they're probably getting them because they're the only ones asking for them, right? I think they've deflected questions about their integrity with the ol' "well, we're not on the take, case closed" routine, which the majority of the public seems completely fine with.

***

Stephen Totilo, MTV News: Many of you were struggling to make a decent argument for the elimination of review scores. Here, I think you just did. Don't put scores on your exclusive reviews in your magazines. Add them via your website two weeks later when everyone else's reviews run. Then everyone's happy.

Or…just don't take exclusive reviews. This would be a challenge for a magazine that needs to secure something notable months in advance of press time. But for websites, is there really a need? You can get the game when it comes out, play through it and have a review up within a week of it's release. No tricky politics. No stink of compromised values. And your readers are served.

The threat of publishers withholding cooperation in the future because of a review strikes me as absurd. I don't doubt that it's a real threat, but there are countless ways to report and edit around any blackballing. Besides, getting blackballed usually makes for a good story.

***

Robert Ashley, freelancer: Just to add to the list of reasons why reviews create editorial drama, it's my understanding that retailers take into account review scores when stocking new games, which means that scores can directly affect sales. I had a PR rep from Deep Silver tell me last week that Target bases their game orders exclusively on Gamespot scores (so, of course, he was bitching about all the snooty divas at Gamespot). I can't vouch for his claim, but just having the guy go off in a room full of other pro enthusiasts noticeably changed the mood of everyone in earshot.

While reviewing for EGM, I felt like there was always barrier between me and PR, thanks to the reviews editor. But when I would venture out to write other stories, visiting studios and going to game reveal events, I would often get snippy comments from people about my reviews. I was once accosted by a developer from the True Crime: New York team about my review of his game. We had, by chance, ended up in the same two-person sleeper car on a train from Paris to Venice. And I was on a family vacation, not some gaming press junket.

***

Dan “Shoe” Hsu, Sore Thumbs: Stephen, getting that first review up the weekend before a game’s out versus a week after launch can be a huge, huge difference in traffic. So most website editors would find an exclusive review very appealing. Can’t blame them.

Those threats from publishers can come in many different forms. Sometimes, it’s just one very specific team or division that wants to blacklist you. For example, we were supposed to get an online exclusive as part of a package deal with our Saints Row 2 cover story, but after they read Robert’s article, THQ told us that the developers didn’t want to work with us anymore. So that online exclusive went to someone else instead.

The Mortal Kombat team and Sony’s sports division banned us as well. But the interesting thing is, these three blacklistings didn’t carry over to the rest of THQ, Midway, or Sony Computer Entertainment. They were very specific to those specific products, because they felt we had it out for them.

Then there are situations where the blacklistings go company-wide, like with Ubisoft after our Assassin’s Creed reviews...and after we wanted to do a story about how outlets were allowed to break the universal AC reviews embargo if they scored the game high enough. That “non-cooperation” on our part was the last straw for Ubi, and when I reported on this blacklisting in EGM, it just further cemented their hatred for me as an individual (even though Sam Kennedy, Patrick Klepek, and just about every other editor wanted to run those stories as well).

Now, I don’t really think Ubisoft or any of those other companies have done anything wrong here. It’s their right to work with whomever they want. But it just goes to show how uncomfortably and inappropriately cozy the industry and enthusiast press are expected to be. You cooperate, you benefit. Simple as that.

***

Kieron Gillen, Rock Paper Shotgun: *Yeah, Shawn, regarding exclusives, on the UK side it's pretty similar. On the print side, perhaps even more intensely. There's just more magazines and the competition has always been fiercer. The former is primarily due to the size of the country - the shipping costs don't cripple you, and a smaller magazine can be more profitable. The second is due to how British games magazines are sold. The vast majority of magazines are sold at newstand, not subscription. Subscriptions are cheaper, but not a straight loss leader for ads. When I was on Gamer, the majority of our money came from actual people buying the bloody thing rather than via advertisers. On the bright side, this abstractly means that in a real way, the bills are paid by people who want to read your mag. On the bad side, since there's an enormous floating readership which you have to fight for every single issue, meaning even more importance is placed on exclusives. An enormous amount of effort is thrown into chasing them, and it can totally lead to the sort of issues others have picked up on.

When I'm chatting to modern day magazine staff, one of my favorite stories to tell is about an old pre-me Future major-mag editor. He didn't speak to any PRs. The Dep Ed did all of that. If any of them tried to speak to him, he just blanked them. He just made the magazine. And when I roll out that anecdote, the look of disbelief which it's always greeted with is akin to what I'd have got if I claimed he was capable of flight.

As I said, they're an enormous waste of energy—I always remember that it seemed that due to the inevitable pissing off of PRs, the exclusives we chased went to our major rival as much as not, and the ones that they went for went to us. But it's also a game which I think magazines—and publishers—are loathe to get out of it. Because as long as one other organ in the room is making exclusives, you are at a distinct disadvantage. And it works both ways. One major games publisher, working on similar logic to me above, stopped doing exclusives for a few year. The amount of coverage suffered—because their rivals were still making deals and they weren't—and they went back on it.

That's the problem with stopping exclusives. It requires a conspiracy of doves for it to hold.

Regarding changing marks and having marks changed... much like others have said, I've done it. It's really not a problem per se. Discussing it with the writer is fine (And an art form in and of itself). Explaining how they're not marking to your mark-scheme is fine too. And most commonly, the review and score just don't match up, where you have to ask that one or the other is changed. If a reviewer is having fun comparing part of the game's mechanics to cancer and still give it in the sixties, either the overwrought writing goes or—if they actually mean the overwroughtness—the mark needs to drop like a stone. In my experience, I've argued far more marks down than up.

On the other side, I'm fine with that too. Sometimes I really don't agree, in which case I ask for my name to be removed from the review. Not in a prima-donna way—just because I don't believe in it, and ultimately, your name is all you've got and you're going to give enough review scores the readers will tear you apart on your lonesome without someone else making them for you. This is one of the first things I say to any writer getting into the business, because—bless 'em—not many are even aware that it's something they can ask to do. And if the editor says no, it's about as good a sign that you should get out of that job sharpish.

And, of course, some editors do change marks without asking and keep your name on it. Just don't work for them again.

***

Harry Allen, Media Assassin: Greetings, everyone.

I just want to say that it's really a privilege to be able to read all of your thoughts. I'm a fan of you all. I read your words and listen to your podcasts, in many cases. So, it's great for me to hear your thoughtfully written ideas about this very trenchant aspect of your work.

I don't know that I have anything to add. Much of what I was thinking has been said by other people, and I know we're about to change topics. For example, a very early concern of mine was the relationship between scores and advertising, so I'm glad that some of that was addressed recently.

I will say that one of the first statements I read that directly connected to my own work was N'Gai's comparison of the reviewer and the critic. Though I use different language, I know that when I started writing about hip-hop professionally, in the late '80s, I made it my objective to never talk about an album in terms of whether I liked it or not. Instead, I always saw it as my job to explain the artist's intent to the readers. I've never called myself a "music critic." I've always said that I am a Hip-Hop Activist.

To me, number scores are mostly an attempt at giving the illusion of numerical precision to functions that cannot possibly bear the same.

For example, take a look at Robert Motherwell's 1961 painting, "Elegy to the Spanish Republic, 70," here: (http://www.metmuseum.org/TOAH/HD/abex/ho_65.247.htm)

Is that a 10? An 8.5?

Most curators and art historians would call the bold canvas "a masterpiece," and leave it, more or less, at that. If they did quantify it, or one of the others in the series, they would only do so in extremely hushed whispers, among themselves, if at all, but never do so for public consumption. This is actually part of the way that they frame their field, which is commercial, but which is mostly a repository of serious, high culture: By avoiding quantifying numbers ("It's an 8"), which would be seen as crass.

Quantifying a Motherwell, then, would be analogous to asking how much the Maybach is: If you have to ask, you can't afford it. They're not saying you can't afford the Motherwell financially, but you "can't afford it" aesthetically. In other words, the experience of the painting is far too rich to reduce to a digit, and if you don't know that, you shouldn't be here.

I think a reason similar to this is also why I always resisted, during that brief period of my life, when male friends would ask me to assign a number to a girl I've seen. In hip-hop / Black slang, a "dime" is a girl who's a 10...but what does that mean? According to what objective scale?

And indeed, isn't that the core idea that disproves the fantasy: That without an actual 10 to which one can point—the theoretical perfect game—the numbers become meaningless?

That is, on a foot-long ruler, "4" only means something because there's a "7," and a "9," but, most of all, because there's a "12." However, "12" only means something because there's a "13" and "25"; an agreed-upon metric, in other words.

When it comes down to it, game numbers facilitate the purchasing decisions of the buying public, and advertising, and I think that's it. I could have told John Davison that his CGW readers would revolt. It's like all those Americans who claim they only watch PBS and the news, yet, somehow, reality shows are a phenomenon, and porn even more so.

But, clearly, the focus on numbers is leading people to overlook something important, namely the subtle interplay of parts that is a videogame.

I've not played Gears of War 2 yet, though I did have the nerve to write about the game's marketing on Media Assassin, here (http://harryallen.info/?p=1863).

But when I look at stills and video from it, and hear descriptions of what Cliff Bleszinski and crew were trying to do, it's clear to me, Roger Ebert be damned, that we're dealing with a moment here. Gears is a work of art expressed in the videogame medium. It's that simple.

Here's a prediction: A hundred years from now, collectors will purchase copies of Gears, Katamari Damacy, System Shock, Crysis, Ico, Dead Space, and every Mario iteration the way many, today, buy coin banks, first edition books, fine watches, and music boxes.

I'm sorry to say this, but, by that point, those collectors will not care if these games were given sixes or nines. They will collect them as beautiful art expressions of human craft and intellect. For many of the artists who work on these games, it will only be then that, aside from the thoughtful critique of people like you, they get their just due.

###

170 comments:

Jomolungma said...

Bless you Shawn. It's gonna take me a while to read through all of this, but I will. There is such a need for this type of conversation in gaming. I think as, sadly, more and more sources of gaming criticism and journalism dry up, it is important for the respected and experienced voices to get together and continue examining and growing the industry. That's why I LOVE Slate's Gaming Club and any podcast that can bring together multiple, informed perspectives on video games. Continue the good work. Can't wait to see the fruits of your labors from 2K Boston as well.

Duffyside said...

I'd give this symposium an 8. Compelling.

Teanaholic said...

Third!

Will Templeton said...

Having only read up to John Davison's second entry, I apologise if this has been discussed in the latter half of the post, but I just wanted to get this out there.

I believe the reason for all the stress about scores is precisely because of that disconnect between reviewer and critic. What games journalists as a whole seem to be doing right now is trying to merge the two, which is a task without merit. Both the ten-point or five star or whatever scale and the critical analysis of a game's content have a place in the industry, but it's vitally important not to confuse the two as they cater to wildly different audiences.

At a base level the job of a review, as was stated here, is to inform a purchase choice. The majority of gamers have limited disposable income. Comparing Prince of Persia to Sonic Unleashed is ridiculous from a critical standpoint but imperative when you only have sixty bucks in your pocket.

The person who wants to discuss how an individual game affects the medium as a whole or storytelling in general or any one of a thousand other factors is a wholly different person to that first audience, and I think that any attempt to appeal to both audiences at the same time is on a path to frustration.

Unknown said...

Thanks for these posts. A good LONG read.

Unknown said...

Although, a spell checker would be nice. There are a ton of typos even in the short pieces.

For example, "shouldn't it that exemplify."

I realize these are emails and you want to maintain the original correspondence, but I keep getting caught up on it. Not trying to troll, trying to give some constructive suggestions.

Sachin Agarwal said...

One of the things left unsaid here is to what extents exclusives are unique to the gaming industry. Apple has a handful of favorite outlets (NYT, WSJ, Newsweek, USAT) who get to review new product, and they've done exclusives (flat screen iMac to Time) before.

We just saw Activision give USAT the exclusive first look on GH: Metallica. To what extent are broader publications inoculated from these pressures? Could not newspapers and general press magazines be able to more accurately do exclusives because they're not as dependent on the advertising from the people they are covering?

I've advocated Tribune Company (here in Chicago) to aggressively go after gaming reviews (either as the CT, the LAT, or a company-wide reviewer) because they can, due to being less reliant on gaming advertising, be more objective even with an exclusive.

I'd love to see your thoughts about whether or not Seth and N'Gai can be freer to do fair exclusives than Shoe or Leigh could at their publications.

-Sachin Agarwal, Dawdle (sorry about the Jerome Walton name; that was my Deadspin nom de plume)

Anonymous said...

Thanks guys. As an avid consumer of gaming coverage, this is fascinating stuff. I guess I can count myself in minority in that I don't much see the point in numerical scores.

The extent to which a score ultimately is useful to me would be a bunch of statements like "buy/rent/don't bother..." broken down with qualifying clauses like:
...if you do/don't like the genre
...if you do/don't like other games in the series
...if you don't/only care about multiplayer/singleplayer/coop"

Jonathan said...

I thought I might help to see what an average gamer/consumer read reviews for. I can typically only afford one game at a time.

If there are a few games out that peak my interest I will go to GameRankings and find the highest reviewed one, read snippets about it and typically go purchase it.

If there is one game I am particularly interested in a I will go to GameRankings and see what overall score the game has. If it is above an 84 or so I will have no issues with buying it. If the score is under that however I do further research by reading the actual review.

If there a few games I definitely want to buy I go through and read every review on each. Like Fable 2 vs Fallout 3, I knew I would play one of them but which to buy first, I had to research to find out. I read everything I can on each and it is then that a review is truly helpful.

If there are two sports games and I have no idea which I want to get (NHL 09 vs NHL 2k9) I will read reviews which typically compare them anyways.

If there is a game out for multiple systems and I am not sure which to buy it for I will consult reviews to help me with my decision.

I don't read "reviews" for critiques, I read blog posts or forum threads or listen to podcast's. That is much more interesting to me. I don't want a critique going over every little thing a review typically would. I want to hear what the writer thought of the game as a whole not broken down into small reviewable sections.

Etelmik said...

The message that I keep seeing here is that it's all about economics.

"Publishers want good reviews. Editors want exclusives. Magazines and websites want advertising. Advertisers want good reviews."

This is the shortest quote that could encapsulate it all. Review scores are simply a symptom of the audience: yes, it is regrettable that a score means people don't read it, but remember (as if needs to be said): we give the Internet and gamers too much credit if we assume they fully read the majority of anything that is posted. Remember the comments in Game|Life's top 5 360 exclusives?

If you want to write more reviews that give little or no consideration to the scores, you need the audience. Is that audience there? If so, how can you get it? The only answers I can think of are economic ones; I hate to be so reductive, but I don't see how business and money aren't what makes review scores go round.

Nate said...

Shawn, I know this is a bit early on, but is there any plans on binding this symposium and printing it? I found this first part fascinating, and I know with much more to come I would love to have a copy of it sitting on my coffee table to make me look intelligent.

John D. Moore said...

Fascinating stuff, and I'm eager to watch as the rest of this symposium series unfolds.

One thing I'd like to see tackled is why the hell game scores are so seemingly inflated. To pull out its easiest analogue, let's look at typical film scores. Rarely do I see as many perfect 10s (or four stars or what have you) for a film, especially when the accompnaying review is a list of things that are terribly flawed about it. From the reactions I see, a 7.5 review score for a game is practically a kiss of death, whereas a film reviewer (like Roger Ebert) can make a case for the value of a 2.5 star film.

Travis Megill said...

Review scores only matter as an aggregate, at least for me. They're a way to summarize a large group of reviews in order to get an idea of what critics thought of it. When I'm looking at a specific review, the words matter far more.

Anonymous said...

Personally, I knew that I'd probably end up sounding like a mix between John D. and Robert A. here.

I'd love you guys to keep this going with all the topics and issues floating around the games industry. It’s an effing goldmine of reading for someone like me.

I'll spare you the long winded personal rant I have on the matter (as you guys spanked it pretty nicely) and just provide some random observations to ponder over.

1 - The idealism that some arguments form here makes me feel a bit snootish for the thoughts that immediately strolled through my head. I consider myself a fairly open-minded person, yet I still found myself sighing at some of the more idealistic based statements here.

2 - "we're all human" got put forth just enough for me to notice it. Not that it's bad or anything, it actually made me smile with a bit of malice in my heart =). Someone go back and count it, it can’t be that many…

3 - I wondered when that damn art debate will come back. Harry's last comments came in like Elika at the last minute for me...They won’t be art until 400 years from now. Seems none of us are smart enough to realistically have it function as a means of existence now. People like Ebert are exempt, as are all people who talk out of blind ignorance.

4 - Psychology and Subjectivity...I prefer to see that to be honest. When those things are exposed, THEN you're really starting to approach whatever sad little reality we hold dear.

5 - Also, acknowledging rationalizations and spurious objectivity is something that surprised me here. It's nice to see the editors knowingly bowing before it (while whispering mean things under their breath).

6 - I’m sure that centuries ago when paintings were the “highest” form of art there was, there was some trendy asshole that walked around pointing at canvases going “OH, THAT’S AN SEVEN!”

~sLs~

Macroe said...

Thank you Shawn for your initiative, time and dedication to this project.

Allen said...

I'm reminded of the proposed Cheap Ass Gamer scale for reviews - if the number is meant to reflect some kind of consumer impulse, review the game on a 60-point scale.

The idea then is that the number tells you at what price you should purchase the game. A game that deserves an instant, day one purchase would get a 60. A game that you should wait for bargain bin status would get a 10. A game which is "good" but not worth "full price" might get a 40.

Arguably, Cheap Ass Gamer is the most consumer focused gaming based portal on the Internet, so assigning a dollar value to a game might be crass... but if that's the ultimate end goal of a score, why not?

Assigning a dollar value on a game, much in the same way that an "expert" on Antiques Roadshow would assign a dollar value to a painting or a coin seems to be what people who want scores are looking for anyway and it unifies the seemingly arbitrary scales that each publication imposes on its reviews.

For what its worth, from an outsider perspective, I don't even read reviews anymore. As a consumer, forums like the 1up Boards or GameFAQs or NeoGAF provide much more useful feedback. If a dozen people in a thread for a game agree that the controls for that game are inherently broken, then most likely the controls are broken. The "non-hardcore" version of this phenomenon is the user review on websites such as Amazon.com. In the "Web 2.0" age of "social networking" and "crowd sourcing", consumers are looking to each other for advice rather.

Anyway, I'm really looking forward to - if it happens - the Game Criticism vs Game Reviewing topic. I'll fully admit that I'd love to "critique" games for a living... unfortunately, the only way to do that right now is to get a PhD and become Ian Bogost.

Das Dorsch said...

I do not think people will buy videogames of the early ages in 100 years as they buy paintings. Paintings were part of a culture that was led by interconnected, highly educated people, elitists if you will.

Nowadays, culture is a thing off the masses, and elitist culture is split into many niches. A far mor appropiate medium to compare this to is the comic book, as it also started as a mass culture, not lead by intellectuals.

On that not, I do not think that games match the narrative experience of the other narrative media. I still love games with a liner narrative, though, because they do things other media stopped doing long ago: they improve in quality as time passes and newer games get made. Stories are getting more serious, immesion-breakers minimized.

Games have become better and better in both technology and story presentation. Because there are such clean, highly visible differences in quality of those divisions, I think a score may be legitimized. Ore so than a movie review having a score, at least.

Unknown said...

http://www.telegraph.co.uk/news/newstopics/howaboutthat/3816785/Performing-monkeys-attack-trainer-in-China.html

Andrew said...

tl;dr

Anonymous said...

Kieron,

Look up tautology in the dictionary. It does not mean what you think it means.

But to be a little more constructive: I was really disoriented by the structure of the responses. Direct questions were asked and never answered (much less acknowledged by the recipient) and whole conversation threads were dropped, only to be picked up again without warning. N'Gai references Harry, and it's half a dozen responses before Harry puts in his first words.

I'm really interested in what you guys have to say, and you say a whole lot in this post worth reading, but I'm wondering if there isn't a more legible way to present it.

Anonymous said...

It’s a shame that blogger doesn't have a more sophisticated comments system, such as only giving comments to specific entries. There’s a large amount that I would like to address on here but it's much harder in this manner.

I do like the "Rolling Stone" scoring method that was mentioned here. It would give a feel of validity to reviews that personally I feel that any score lacks. Personally I don't trust review scores at all, they work on a comparative scale that differs in each reviews outlet, although I do have greater faith in the new 1UP grading system, which is a much more general indication of quality, (rather than a specific indication of a less understood scale of quality). When I read reviews I've always read it in its entirety to see what the reviewer thought, usually in order to see if it was a game that I would like and also to see if there was anything that would put me off wanting to play it, often reviews would decide whether I brought a game that I wasn’t sure about.
I think that review scores cause a lot of problems, while they aren’t inherently evil they are quite possibly the root cause of stagnation and corruption for video game reviews based industry, possibly even the games industry itself. While this scoring system is in place PR will remain fixated on them, trying to influence and even control the outcome of scores. Sometimes this will work and even if it doesn’t it will often mean relinquishing exclusive reviews and as mentioned above magazines need to co-operate to succeed. Since many publishers now are internally ascertaining the quality of a game and the ability of its design team by its Meta critic score, in a worst case scenario this would mean that the actual content of a game is irrelevant it will receive a certain score regardless. It would also mean most reviews might as well post a number and add some text to make it look official, maybe the games press description (a review Shawn mentioned some time ago actually did this, maybe he’ll link it in a later topic).

I believe we have this bullying of reviewers in video games but not so much in other mediums such as film is due to the youth of the industry, and also of its audience. Games are not yet universally understood and accepted as films are, they don’t have the wide spread and varied audiences quite yet, although this is changing at a exponentially increasing rate, mainly due to Nintendo. I don’t believe there is yet a large enough demand for high quality reviews or mentioning of certain types of games in non-gaming publications. Such sports games reviews in sport magazines and critiques talking about the “best entertainment experiences of 2008”.

Video games aren’t to the point where a review could be in a general newspaper and most people will have a general understanding of what you’re taking about. Because when gaming reaches that benchmark when it’s reviewed as any entertainment would be to a world that thinks of games as a regular entertainment, that will be when specialist press will need to either become where you would look for high quality gaming reviews, or appeal to specialist or niche audiences within gaming.

Mike said...

Really need a print version of this.

class of 89 said...

Holy shit. Erickson was right.

Jake said...

Thanks for the post - this is a very interesting conversation. I'm looking forward to reading future symposiums.
Jeff's appraisal of review scores' purpose as shortcuts for an audience unwilling to think critically about a reviewer's prose seems like a bigger discussion and, possibly, the real root of this conversation.
Is "games journalism" shackled by its history and its younger audience? Why is it that thoughtful conversation about games is much more likely to be found on blogs like this and in British gaming publications?

Juan Letona said...

What does one of the "normal" people (as Dan Hsu describes us) feel about review scores? From someone who has read reviews since 1989 for purchasing decisions to now, my attitude has changed

At first, review scores have become the litmus test of a game that I am on the fence about or now nothing of. I want a review to answer the questions of a cynic, a skeptic or just a confused gamer if the game is worth one's time. It is especially true for sequels, sports games and now, all those Activision games that get once a year treatment.

Reviews are no longer the only source that influences my purchasing decision, I often listen to podcasts, the vidcasts, demos and/or follow the "buzz." Even previews have time to time influenced a decision, just recently, Shawn Elliott's Far Cry 2 preview spark an enough interest to buy the game. Stephen Totilo's daily dairies have become like a book club, as I play similar games, for example, I experience the same frustrations that Stephen faced in Dead Space.

Reviews will never go away, as Dan Hsu points out, "It’s ingrained in society and it’s pointless and stubborn to fight it. People don’t always have time to read a 2000-word, well-crafted review to get inside the brain of the reviewer. For most folks in this short-attention-span world, that “4 out of 10” usually says more than enough." The problem with Review Scores are not the scores themselves, but the fact that one person can never review the bulk of weekly game, as Roger Ebert does with the movies he watches for the Chicago Tribune. The last point is better reserve for the next topics.

TenaciousCK said...

Reviews for the most part are completely pointless, at least for the enthusiastic gaming consumer, which I'm sure anybody on here reading this is. I don't need your review to tell me that Metal Gear Solid 4 is going to be worth playing. I also don't need your review to tell me that Grand Theft Auto is going to be amazing as well. To be honest, the only time I look at reviews are for the games that don't have the marketing of these mega titles. In this day and age where games pretty much sell themselves, I rarely if ever need a review to merit my purchase.

Then of course there's the possibility of being burnt by the review process. Far Cry 2 got a B+ from 1up, 90+ scores from other sites and what not, and I ended up really not liking that game much at all. What I want to see in critiques is a critic's ability to point out the design of the game, how it compliments its other parts, what certain decisions of the designers worked, what didn't. Why the game was designed the way it was, and how it affects a persons experiences of it. Mainly I just want to see critics question the way the game was design, instead of simply absorbing it and reflecting it back to consumers. Maybe this is a lofty claim, I don't know, but I rarely if ever seem to get this. I do think reviews are needed in this industry, but I just wish people would lend an ear to critiques of games as well.

Why can't we just have both? A review score for a game upon its initial release to the market, then when its shelf life has diminished and there's a sense of detachment from the product, then the criticism can kick in. Idk, perhaps this logic is backwards but I think after the consumer has had a chance to play the game, perhaps constructive conversations can then start on message boards. People just invest too much in a titles release, they need time to sit on it just as much as reviewers/critics do...

Anyway, hope that all made sense...

trip1ex said...

It seems to me the internets makes this discussion a bit irrelevant (albeit more possible and more practical.)

I'm skeptical along the lines of Totilo.

There's plenty of more in-depth & "alternative" coverage of games to be found on blogs, forums, small sites and podcasts.

For anyone wanting that type of content you can find it. Or you can create it.

So it doesn't seem to matter if the big mainstream sites & mags are heavily score focused, publish nutritionally-lite reviews and broker exclusive reviews, etc.

Nowadays, thanks to Al Gore, the thirstier of us gamers have plenty of alternatives.

Colin said...

Thanks so much for a thoroughly deep and fascinating discussion.

There was one point touched upon that still left me thinking though... there seems to be a dichotomy in all this between writing for the enthusiast audience and writing for the "average" gamer (not casual just... less hardcore). The purpose for the score, in my experience, is taken differently by these groups. Hardcore gamers are the kind you generally find on message boards defending and refuting the score but, as mentioned, have probably already pre-ordered or will play the game regardless to be part of the discussion. The gamer who is less invested in being a part of the society of games discussion/review/news, however, will probably use it as an aid to his/her buying decision, if they are aware of it at all.

If this is the case, the score plays a dual function of ranking the quality of the game and recommending the game for purchase, which I don't believe are the same thing. It's little wonder then that they provoke such a wide range of opinion. If a writer speaks to the value of the game for the kid who buys 2-3 games a year they ignore the use of the score by the person who buys 10 games a year and doesn't need that return on investment from every game and vice versa. Certainly these two goals can overlap but I don't think it's possible to consistently serve both masters.

By the way, I almost never comment on blogs or reviews but the quality of this symposium, even at this early stage, is so satisfying I couldn't help myself. Looking eagerly forward to the next installment.

Jonny said...

"which makes me wonder if we shouldn't be trying to create more accessible, readable text, but that's probably a whole 'nother issue"

Yes.

Jonny said...

"Unless I'm missing something, that's quite an indictment of a portion of your audience. Why would you write for people who won't or can't read an entire review?"

Because those people generate ad revenue, which pays wages. Do you live in the real world?

Not trying to offend, but really, we don't all have the freedom to write thoughtful stuff for people who actually appreciate it. Someone has to write for everyone else.

Jonny said...

Sorry, I should also add that despite my reservations, I did find this very interesting and I'm looking forward to the next part.

Justin said...

This symposium reinvents the compelling wheel and truly is greater than the sum of its parts. However the graphics are last gen and there is no sound. 10/10

:P

Just wanted to say thank you very much to everyone who is participating in this. There is allot here & I am going to read through this again.

Unknown said...

I hate scores because they're too easy. Being able to cross-refrence a 6 vis a vis a 7 is a comfort, I submit, *only* due to it's immediacy. There isn't any subtlety, philosophy, or art applied in strictly numerical evaluations (though Gillen's thoughts on 100-point scales is incredibly thought-provoking for me) and while I can't for a moment dismiss the commercial applications of a publication which focuses on scoring reviews I yearn for a space where the paragraph replaces the digit. I think a continuing sign of a medium of expression in it's adolescence (I think we've at least moved out of our infancy) is the lack of a truly viable space in the likes of what Harry Allen referenced. There's an intellectual and--dare I say--spiritual need for such an endeavor and that is where I see at least one of the "problems" Totilo is asking for; right now we've *only* got review scores. There isn't an adequate counterweight.

Shawn Elliott said...

I appreciate all responses. They will inform future questions.

Laez said...

That took a while to read, and after burning my brain on that all I can really say is, fantastic.

I wonder, though, what significance you all see in the increasing popularity of podcasts and their influence on reviews and what readers take from them. Seeing as most of the contributors to this symposium regularly or at least occasionally appear on reputable podcasts, I'm sure there's a little something to be said. As for myself, I've found that in the past year or so that I've more and more been basing my purchases what's being talked about positively on my favorite podcasts. If a game is on topic for weeks surrounding it's release, and is constantly being raved about, that means massively more to me than a 10 ever will, and by the same token when a game is regularly mentioned I'm more likely to check out a few reviews even if I would have otherwise dismissed the game.

I don't know if I'm alone in that, but if I'm not I'd like to read what the symposium-ers (Is there a word for one who contributes to a symposium? I'm having trouble finding an easy way to refer to these fine folks.) think might be the reason behind that (I've got an idea or two about why it's so in my case) or whether they place any importance on podcasts at all for that matter.

I look forward to the future postings.

Anonymous said...

My favorite part of the above unreadable wall of text has to be everyone avoiding the obvious answer. People want review scores because any halfway decent writer wouldn't be writing about videogames for Christ's sake.

The job consists of putting a number on a review, you should have a 10 in 10 chance of getting this right before dedicating your time to fart/boob jokes on the screen captures, yet some still manage to fail. How often can you write "it depends on your tastes" (spoiler: every time) before you sit back, stroke your chin, and look at your legacy of awful reviews before just deciding to finish college/snap and start killing people?

"I hope there's an anecdote about the reviewers terribly interesting life in this one!" Do you recognize this quote? That's because it's never been said, ever. Fight through the tears, think of how valid an opinion on something you can't be bothered to finish is, and just give it a 7 already.

If someone wallpapered my house with this I would burn it down before reading it.

Unknown said...

Dear sir, may I recommend the following past time? http://i42.tinypic.com/2yod4dv.jpg

frrrr said...

I feel that reviews are generally too eager to gush, mostly because they are written around the same time as the game itself.

It makes sense to do it that way, but think of all the cases where the hype has died down. When people have looked back at Bioshock and gone, “You know, maybe it wasn’t the most important game of all time” or Oblivion which went from “Oh my God this game is sooooo amazing” to a whopping about-face “Holy shit this game is terribly broken and emotionally sterile to its very core” It’s this weird nerd high that goes around. The same one that had everyone convinced for about a week that Star Wars Episode 2 was a good movie.

9/10s are handed out like candy on Halloween. Somewhere along the line it all just becomes a blur of hyperbole. Plus, and no offense, but most of the time the writing is so bad and obvious that one glance at “9/10″ and you can predict the details through intuition so what’s the point of reading on? Oh, you say it’s really really good but has small flaws in a few key areas that keep it from being perfect? Brilliant!

To tell the truth, the only people who really, genuinely care about reviews are the doofs who argue about it on the internet and are worried only about having to do damage control over a low score and thus protect whatever sliver of self-worth they get from that. I’d say far more “real” people get their purchasing decisions from personal recommendations than reviews.

I’d greatly prefer games to be “read” instead of “reviewed”. Something along the lines of a dissertation on how moving the Fallout series from an objective, detached camera to a subjective one has all but ruined the subversive atmosphere. (Which is true, btw) It’s just something you can’t sum up in a tidy score and does a much greater service to the people who read it.

Unknown said...

Thanks for a most interesting read.

Regarding scores on reviews, I often use them along with the reviews name to decide whether I’m interested in reading the text. The score is often an indicator of what to expect and to be honest it’s when that deviates from an average when I’d be interested in reading. I’ll definitely agree with Kieron’s point about hyped games being 9/10 and then losing points and others starting at 5/10 and gaining them. I use that metric internally to decide whether to read. Learning what it is that would make a game lost those points or conversely gain them is a matter than can only be learnt from reading. Seeing the expected score can make even a small review a chore to read. I didn’t feel the need to read anything many reviews about GTA IVs 10/10s, 5/5s or A+s . I was expecting a gushing report about how everything was amazing and it might be flawless.

The other problem with reviews like those, is that coming back after the fact the reviews, to me, felt incorrect. Personally I’m not a massive fan of GTA IV. The opening moments and up to half way through I was with every reviewer but after that I felt the game lost its way, dragged on and stopped being fun. The only place I found this sentiment I felt was in the GFW podcast by Shawn Elliot.

Podcasts are where I get the majority of game opinion from. Those “what we’ve been playing” sections are great for getting a feel for a game and the reviewer. As I mentioned at the start the reviewer is key for me to start reading a review. Learning what things interests any of the reviewers and how their viewpoint compares to mine influences what I believe their score or text could mean to me. Review text though, would be an extension to the discussion given about a game on a podcast.

For example, learning how Ryan and Jeff of Giant Bomb were put off immediately by the first person perspective of Mirrors Edge meant that I knew I personally wouldn’t be able to agree with their assessment of the game as our views severely differ.

I use podcasts for learning about a game because I can do it while I’m doing other things, because there will be a discussion between people about the game and because you don’t need to put a score at the end to know what the person is thinking.

Chris "Papapishu" Person said...

Shawn and N'gai touched on this briefly but I think that on some level this critique misses the forest for the trees.

First, listen to this:

http://feeds.wnyc.org/~r/radiolab/~5/143890856/radiolab081407pod.mp3

I was listening to Radio Lab a while ago. They had a great segment on emergent thought in ants and humans. Essentially, if you look at an ant colony, there is no leader. It's braindead. And yet if you get a bunch of ants together, they're suddenly smart. Individually they are stupid, blind insects. Together they do infinitely complex tasks that most humans couldn't do.

The same thing, the program explains, happens when you get a bunch of humans together. If you get enough people in one room, ask them to guess how much an ox weighs or how many jellybeans are in a jar you'll find some thing funny; The average of a lot of stupid guesses is more accurate than any educated guess made by an individual.

So in terms of game reviews, the group may be far more knowledgeable than any single person. I'm not just talking about journalistic groupthink or the metacritic score either; I'm talking about kids yammering about the DS on the playground, about the recommendations of minimum wage EBGames employees and about the chatter of people that haven't owned a console since Atari. The value of a game is not determined on a microcosmic level but a macrocosmic one and at the end of the day we shouldn't worry about a review by in isolation, we should worry about the discourse we're having and what influence that discourse has on the group as a whole.

So despite how much I love reading thought-provoking reviews, we need to realize that this is not a solipsistic medium - this is a choir of opinions leading ultimately to a conclusion. If we're going to get anywhere, we need to be conscious of that.

But then again, what do I know? I'm just an ant.

Matt Jett said...

As a total wannabe, writer, this conversation is inspiring on several levels.

I think most of the issues involved with reviewing video games come back to the idea of reviews being treated, by dev. studios, game publishers, or magazines, as total commodities.

The utter lack of transparency that most outlets have regarding the hoops they have to jump through to obtain exclusive reviews contributes to a severe culture of mistrust towards those reviews among those informed about the review process. Sure, we know it's an unfortunate necessity, but that doesn't make me wonder any less if those early scores are valid or simple money-makers.

Ultimately, though, the scoring process is just one symptom of an entire culture centered on game reviews that needs to be overhauled. So what if you take a 10/10 score off a game that doesn't actually deserve it, but got one based on the fact that's is a hyped AAA title? Does that mean that the review is suddenly more valid, that the flaws glossed over to give it a 10 are still not ignored in the text?

I don't think the problem ultimately lies in what scores games are given but rather in what appears to me, as a semi-well informed reader, to be the process behind arriving at those scores.

I know this reads like a long-winded way to say "scores aren't the problem, you guys just write shitty reviews," but that's really not what I'm trying to get at. It just seems to me that, for whatever reason, a lot of games are not judged equally. This is touched on by you guys in your discussion about having preconceptions before you begin reviewing, but it seems to be way more prevalent in reviews than the symposium makes it out to be.

Oliver Snyders said...

Holy Holy! Fantastic read (and I'm only halfway!).

Everybody balances everybody out. Will continue reading ASAP!

Doogie2K said...

Stephen Totilo, MTV News Multiplayer: Have any of you come across gamers who won't buy a game they were curious about because you gave it a 7 and not a 9?

Well, speaking as someone who doesn't buy many games at $60 (the theoretical target audience of reviews), frankly, if a game I'm curious about gets a 7, I'm not gonna buy it for $60. Instead, I would wait six to twelve months, buy it off the bargain shelf at EB or Future Shop for $20 or $30. Crackdown and Assassin's Creed both had flaws that would probably bother me at $60 that I really didn't care about at $30, because I still felt I was getting my money's worth.

I think that's really what I look for in a review: am I going to get my money's worth? And in a lot of cases, I approach it from the perspective of how much gameplay I'm going to get out of it. Portal was one of the best three-hour gaming experiences I've ever had, but if I'd paid $60 for that alone, I might have felt ripped off; for $15 or as part of the Orange Box, I think I actually got more than I paid for, because of the fantastic writing that backed up the solid gameplay mechanics. At the other end of the scale, ESPN NHL 2K5, Burnout 3, Guitar Hero III, and Diablo II are all games that I've sunk hundreds of hours into at various times over the years (to say nothing of the almost two years I lost to WoW), and one of those was $30 off the bat. I'm more inclined to pay $60 for something that I either know is going to last me a long time, or that I can reasonably assume I'm going to enjoy immensely, based on past experience with the genre or developer*. Games that get a 7 are games that I may be interested in trying out because they sound interesting (or because they're from a genre/developer I know/like/trust), but because of my student budget, I'm just not willing to invest in as an early adopter for fear of disappointment.

It occurs to me to wonder now if that makes me part of the problem: I tend to avoid online forums like the plague, having passed the age of 14 many years ago, but maybe I put more stock into a review score than it really warrants. That being said, in recent months, I've gotten into a number of gaming podcasts (Giant Bombcast, 1UP Yours, 1UP FM, GFW Radio/LAN Party, Evil Avatar Radio/In-Game Chat, Australian Gamer Show), and the more in-depth discussions that occur there have changed my perspective a bit on review scores and what I might like to invest my minimal funds into; besides, it's easier to listen to a podcast on the train than set aside fifteen minutes to read three or four reviews, and you often get more fleshed-out, "real" perspectives by way of interaction in the process. To borrow Totilo's own example, Too Human got excoriated by the press for its repetitiveness, awkward controls, unfulfilled story concept, and numerous gameplay delays/complications (the 20-second resurrection animation and the menu system) but the guys of EAR, as it was known at the time, all loved it for its co-op aspect and loot-whore reward system, and it occurred to me that if it was basically a sci-fi Diablo clone, I might like it, too, even if I have no one to co-op with myself. Because of the differing opinions, I'm inclined to download a demo or wait for the price to hit $30 before buying it, but I'm more open to the concept of a purchase than I might have been a year ago, which is something.

So to bring it back to the subject at hand, I wouldn't say I'd never touch a game if it scored less than an 8 or 7.5 or whatever, but I'd definitely be more wary of investing a lot of money into it for budgetary concerns, and that the "shit filter," as one person called it, does work to help me choose between games I'd buy for $60, games I'd buy for $30, and games I wouldn't buy at all. As long as it's understood that that's what the score is for, I think I'm fine with using it as a proxy for that sort of decision, in the most basic sense, but at the same time, nothing can replace an honest-to-goodness discussion, and I do think a lot of people tend to avoid that discussion because it might assault their preconceived notions (though that's no doubt a subject for another day, and one that's far from gaming-exclusive).

* - Ignored in all this discussion, for obvious reasons, is the store-packaged deal: I got Doom 3, Knights of the Old Republic, and a SoundBlaster Audigy XL with Hitman 2, all for $70, which is what Doom 3 would have cost by itself in Canada at the time, all for being one of the first ten people into Future Shop that day. KotOR turned out to be the winner of that four-item package, but how was I to know that at the time? I went there to buy Doom.

HOSB said...

Regarding the "Rolling Stone" rumor mentioned numerous times here. I remembered this entry from veteran rock critic Robert Christgau's website:

http://www.robertchristgau.com/xg/cdrev/kimya-rs.php

In the postscript, he explains how a four star review was changed to a three star review by the editors of Rolling Stone, but his tone makes it sound as if this was a rare exception. At least as late as 2002, reviewers were submitting their own numerical scores for that publication. So, uh, there you go.

Unknown said...

Nice work, just reading through it now and may comment later. Just a note though, could you add some anchors to the ** you've put in so I can bookmark my position as I dip in and out of this.

Shawn Elliott said...

Great point, Papapishu.

J.Goodwin said...

Well, now I've read this pretty much top to bottom.

I have to say that I right out disagree with N'Gai's position that scores are harming games media, and I've decided that I also disagree with the degree to which certain people (particularly Dan) are getting worked up about this issue as a whole.

Frankly, I rarely read reviews anymore. I pick and choose posts to read off of Joystiq and Giant Bomb, subscribe to a few podcasts (Joystiq, LAN Party, Retronauts, skim 1Up FM and I listen to episodes of 1Up Yours until Garnett gets liquored up and goes apeshit yet again). I might even download a demo, but that's getting pretty rare as well.

I feel like I'm still getting a reasonably good feel for what is going to be good and what isn't. I can't exactly explain how, but there's an overall tone of popular discourse aside from whatever might be happening on "popular message boards" (which I don't read) that fills me in on what I should be paying attention to.

I've also found that in lieu of relying on scores, I just "buy-in" at a price point that's more appropriate to the level of risk that I'm willing to take on a game. If I'm not sure I'm going to like a game, I buy it when it's available for twenty dollars (Army of Two). If it's a sure thing (in my mind, a good example was Rock Band 2), then I'll shop around, but I'll still pick it up at launch.

I wonder if the elephant in the room is that no one reads these reviews anymore anyway, except juveniles, marketing people, and other videogame reviewers.

I couldn't care less what you reviewers are doing, I'm not relying on you to tell me what's good and what isn't in your reviews, I'm getting it from everything else you do and say.

blackwax said...

Great discussion so far. Not sure if you meant for this to turn into a sprawling discussion about review scores in general, but I think it needed to, at least initially. I'd love to see a panel come from this, even though there's no appropriate venue for that right now.

To add my own 2 cents, I don't think you guys will ever find what you're looking for. Several different notions of what functions these scores perform emerged throughout the discussion, but the one thing agreed upon is that people need a bottom line. Consequently, publications need to provide a bottom line. The only question is, how can this be justified?

It seems like reviewers want to reconcile meaningful scores with thoughtful review (and and an audience who wants thoughtful review) within the context of a feasible business model. Do we have any proof that such a notion is anything more than modern day alchemy?

I'm personally a fan of many of the people involved in this symposium, and I appreciate all the thought you put into your reviews. But it's been made clear by Metacritic, pressure from developers and publishers, and responses from the enthusiast community that overwhelmingly, the most important part of your job is the score you assign to games. Truly shitty, but perhaps in this case, it would just be best to let sleeping dogs lie. You are paid to contribute your thoughts on an exciting medium. If people only care about the score, then they'll get exactly what they want.

Nicholas James West said...

On accident I posted this query on the previous blog, but I was wondering:

What would the discussion be like if a bunch of movie critics round-tabled on the subject of review scores?

I appreciate the distinct personalities and sharp insights.

Anthony said...

I was wondering if you could get Jeff Green in on the next part of this symposium. I think he was serious about wanting to be a part of this on the GWJ podcast, Shawn (yes, I listen to way too many podcasts and know entirely too much about everyone on here.) I think it would be interesting to hear him chime in as well.

Marc said...

Having only read the first 3 so far... I figured I'd throw my 2 cents in... in point form, because no one could be bothered to read this anyway.

-scores suggest infalibily, in so far as they are set values (9 or 10 out of 10 is not debatably in its worth). People percieve numbers as facts. Letter scores are automatically converted to numerical representation in the brain.
- because of the previous, the review gains exclusivity and authority. What should be an opinion becomes a fact: game x is 9/10, so how can 9/10 be 6/10?
-a term paper is marked by a lecturer with a degree or three, an olympic gymnastics competition is judged by professionals; yet any tom, dick or harry can pass out a game score-> anyone can claim authority, and because the world of the internet is grounded in anonymity anyone can claim expert status.
-Print represented an expert status in games. If you worked for CGW your talents and discression were vouched for by the magazine. On the internet everything is temporary and only valuated by how many clicks it could get, and controversy gets clicks.

Im rambling now...
for clarification, I study Arts at Sydney University; majoring in political science, sociology and digital cultures. So while my bullshit may be bullshit, in RL i could at least back it up.

Shawn Elliott said...

Marc, we certainly are reading every comment here. Thanks for your feedback and thoughts.

Nicholas James West said...

Marc, good insights but you said:

"a term paper is marked by a lecturer with a degree or three, an olympic gymnastics competition is judged by professionals"

and we must remember that "professionals" still have bias and personal opinion. A degree or job doesn't really make their insights more poignant or real.

So there's a question: What constitutes a professional in the gaming world?

Allen Goode said...

Just a quick point. I like video game scoring and despite its problems I find them very useful.

Sure they are inflated by the various measures stated by the panelist which is why sources like metacritic are a great moderators.

I personally use game scores as a Table of Contents. You see I'm still going to purchase certain game whether the press gives them good scores or not. My own opinion is just as valid as theirs, in fact its more valid because they aren't me and likely have different tastes. But review scores allow me to discover games I wouldn't normally pick up.

Not having a score would mean that I'm not going to read your review for "Generic Game Name Here". However giving it a score shows me instantly that this may be something I might be interested in. I will then read your review, read other publications reviews and watch the video reviews.

Honestly if you didn't post a score I'd likely not visit your site. I am looking for a gauge on the products experience (as you saw it), and at that point, if I'm interested, i will read your opinions because that's where the real heart of the subject will lie.

Bib said...

Thank you for this very impressive and valuable collection of thoughts.

Unknown said...

Shawn, thank you for assembling such a concise yet varied discussion of this topic, which has been so ill-researched in the past. Thanks also to the contributors, whose interplay makes this feature diverse and thought-provoking.

That said, I find myself agreeing most with N'Gai: reviews do not need scores. When we look at the assignment of reviews as an art in its infancy and don't hold fast to established norms, it becomes clear that the advantages to eliminating numerical designations across all critical outlets are significant and plenty.

The issue of garnering exclusives or pleasing advertisers fades immediately; if there is no competition based on scores whatsoever, coverage will be given to whichever outlet the publisher deems most appropriate to reach the game's intended audience. This has the added benefit of encouraging the largely homogenized gaming press to specialize, be it by genre, platform, or whatever else divides the gaming public (Mitch Krpata's taxonomy of gamers comes to mind).

What's more, freeing reviewers from assigning scores allows them to be candid and straightforward, rather than sugarcoating their criticisms to fit an overly inflated score. The fun-to-read very high and very low reviews are evidence of this problem: in these situations, the reviewer doesn't have to please both his own conscience and the public, resulting in sharper opinions and better writing. The in-between reviews are littered with this conflict, and removing scores from them would solve this problem cleanly and efficiently. If the reviewer truly is conflicted about a game, let him say so, rather than trying to reconcile his beliefs into a number. In this way removing scores will improve game review writing, and give it a greater significance to the non-gaming public.

Of course, the implementation of such a system is incredibly difficult, as every sect of the business has become dependent on scores in some capacity. What is required is an outlet free of commercial obligations, so as not to be bound to please buyers instantly and unconditionally, to drop scores altogether. Theoretically, the improved writing quality will bring prestige to this maverick outlet, and others will emulate. It's a dream, yes, but I think one worth believing in.

Dan Kaplan said...

What an excellent and thought-provoking discussion; I commend you all for your participation.

Yet in this entire discussion, there are two absences which I find troubling:

-The comments offered by the primary participants are all originating from the same basic perspective, that of a reviewer. While these insights are varied and interesting, I would have liked to see some representation from the other parties involved: a developer, a full-time PR guy, a common reader. What are those people looking for and how do they view review scores?

-Tom Chick (and others) mentioned the "context" under which he writes reviews. That context is obviously unique to every reviewer, as it is a culmination of all past experience and personal prejudice.

Yet there has been no mention in this entire discussion of the simple fact that the /reason/ a single review score is an innapropriate expression is that it is devoid of all context. Did the reviewer give that racing game a 7.0 because it was buggy or did they perhaps just prefer a different genre?

Since there is no way for the reader to determine /why/ a game is scored the way it is (short of reading the entire review), there is no way for them to evaluate on score basis alone whether or not that number holds any real meaning.

(Case in point: I saw a Metacritic score for Assassin's Creed and elected to hold off my purchase; later I read some reviews and found that while the scores were low, the positives of the game were right down my alley and the negatives were for things I didn't really care about.)

Additionally, it is clear from all the responses on this issue, that reviewers clearly each have a different focus when it comes to actually giving a review score.

So my question for everyone is this:

What can be done to to create a consistent & concise rating system from which all parties can derive meaning? Or is it futile to even try?

Andrew said...

[Disclaimer: I realized that after finishing writing this comment, I appear very score focused, but this was not the intent. I am merely concerned with certain trends very prevalent in games reviewing. This post has nothing much to do with games criticism, which is a separate topic.]

I think a major problem with videogame reviews vs. reviews in another medium, such as film, is that there is too much consensus between different outlets' videogame reviews for me to take them seriously.

There are few movies that garner truly universal acclaim or get consistent scores for nearly every major critic yet in games, it seems to be the rule. Amongst film reviews, I can almost always find a reviewer that liked or disliked a movie for many of the same reasons I did, but with videogames, this is quite often not the case. I don't need or necessarily want to just read someone's restating of my own opinion, but I find it very strange that not one professional reviewer exists who feels even remotely like I do about a game.

For example, a quick glance at the review scores for Little Big Planet reveals that no critic among those aggregated on metacritic gave the game anything lower than an 8/10 or equivalent. While I generally object to metacritic style aggregation, this seems absurd.

Regarding LBP, there a quite a few people on internet message boards that I frequent as well as people in my group of friends who play videogames, who have expressed dissatisfaction with the game similar to mine: the core game isn't that interesting, and no amount of level creation or level editing will amend that, and yet despite being a commonly encountered criticism of the game, no professional reviewer seemed to have these issues.

It's things like that that make reviews so suspect at times. If the gaming community has a full range of hypothetical scores they might award a game, why do professional game reviewers' opinions not reflect anywhere near this range, especially for certain, very high profile games?

Papapishu makes an interesting point, but it still implies that aggregation of opinion will yield some kind of scientific fact, which for all his interesting points about group behavior, is absurd, no matter how it is stated.

Sometimes reviews truly seem to encompass the full range of opinion; divisive titles like Mirror's Edge or Assassin's Creed being such games, but when that sort of range is the exception, how can I possibly rely on reviewers?

Hunter said...

It seems to me that the problem in reviewing games is the inherent subjectivity of the game's quality. I know that there are certain games that, for one reason or another, I can't get enough of, but if I think about them critically they wouldn't get a "10" from any sane reviewer. I have probably logged more hours in Puzzle Quest than any other game, but it does not have nearly as much time, art or storytelling put into it as, say, Half-Life 2, so which one is the "better" game? I would say that Half-Life 2 is the better game in terms of gameplay, but Puzzle Quest is the better game in terms of replay-ability and mindlessness. Shawn Elliott can not say enough good Company of Heroes and, while I respect his opinion, I know that CoH is not my kind of game so his opinion is moot. Because of this subjectivity reviews must rely on seemingly quantifiable aspects of the game. Graphics, sound, controls, etc. The problem is that, in the end, these aspects don't make or break the game. As gaming technology gets better it is expected that the graphics will improve, but if the antialiasing is a little janky, if the fingers of a model aren't articulated, if the blood splatter isn't realistic, does it actually affect how much fun the game will be? The answer should be no, but there are gamers (and reviewers) who take minor graphical flaws as personal affronts. Clipping is tantamount to punching their dog.

I really enjoyed the conversation in a GFW podcasts between Shawn Elliott and Sean Malloy about BioShock. So rarely do gamers get an intelligent conversation about a game, rather than a one-sided review that usually boils down the graphics rather than the content. Shawn and Sean discussed things that I had not thought about while playing the game and, on the next run through, I was more aware of the underlying themes and perhaps even a bit more critical of the binary good/evil aspect of the game.

I think what should be important to gamers and reviewers is how much fun, thought inspiring, or interesting a game truly is and not any so-called "niggles" that can be found. Attaching numbers to a review is arbitrary and cannot be considered a valid reflection or substitution of the review's content. Scores are the TV dinners of game journalism, they may satiate a basic desire, but they offer little real nourishment.

Gabriel R-T said...

I wrote a blog post influenced somewhat by this discussion i think its pretty interesting but who knows maybe it suck...check it out
http://www.1up.com/do/blogEntry?bId=8976319&publicUserId=5753283

Jon K said...

I treat scores as indicators of the value (to me) of the reviews they accompany, and not as a means of judging games. They could be replaced perfectly effectively with a recommendation system, showing me the reviews I might like to read, without reference to a number.

When reading a magazine, I look at the scores first and will jump to the text for the reviews with a high score, or with a score that differs from my expectations. This helps to focus my attention on reviews that might have something new to tell me about a game. Scores of 10/10 for a game I am certain to buy, or 6/10 for a game in a genre I don't usually play, will not catch my eye.

~ george said...

much reading was done, and still the question remained unanswered... why so low?

Adam R said...

I've got a bit of a dissenting opinion as far as this Symposium goes. I can appreciate the overarching focus, but I think the methods are somewhat lacking. I've posted some of my thoughts on my Wordpress blog; rather than reproducing the entire thing here in the comments, I think it would be best to just provide the link. I'm curious to hear what some of you participants think in particular.

Blog link

Luke Lane said...
This comment has been removed by the author.
Unknown said...

Shawn Elliot, you've raised the level of debate and increased the amount of intelligent discussion around gaming. Thank you.

BJ said...

I was going to write a long comment but it was fillied with stuff everyone here already knows. Instead I'll just say thanks for the virtual symposium. I found it far more interesting to read this discussion than almost any other videogame related article in recent memory.


Someone call Leo Laporte and get advice on a virtualized podcast similar to This week in Tech but starring these veteran's of the gaming world and I'd listen religiously. I might even pay a tithe.

Adam R said...

Where do you Symposi-ites fall on this:

Critic in Exile: Is It OK to Finally Admit That I Didn't Really Like Fallout 3 All That Much?

Bobby K said...

That was a very interesting discussion. I love peeking into the mind of a games journalist.

At the age of 40 and still a gamer (although I don't have the time to devote to it that I'd like) I will say review scores themselves are less important to me but at the same time I want to see them. If for no other reason than sometimes it gives me a reason to dig down into a review to see why it got this score.

Case in point was Tom Chick's review of Red Alert 3 on Crispy Gamer. I wasn't even looking for the review but came across it and decided to read it because I was curious about the "Fry It" score. It seemed other sites and magazines were rating it pretty high from just casually looking around. So I read his review and very much enjoyed the depth Tom got into and not holding any punches on his opinions of the game and its flaws as he saw it. I can't judge the review other than to say I think he backed up his score. I think that is the best and ideal relationship between a score and body of a review. Can you back it up. Especially given a scale like Crispy Gamer's. A reviewer can't fudge things. There is an obvious demarcation between Buy It, Try it, Fry it. That is a scale that works perfectly for me because I feel it makes me want to read a review to see why it got that grade. I've gotten to where I hate the 10 point or 100 point scale. Even the 5 point. I think 1UP made a great decision to go to letter grades. I prefer that or Crispy Gamers scale to anything else out there.

In the end though, who cares about the score. I mean really, as critics and journalist, what I want most from you is the text. The in depth review of a game either as buying guide or as a critique. They can either be combined or seperate things.

What I would love to see is an outlet that does an in depth critique and investigation into a game after a game has come out. After the review/buying-guide. Pull no punches. Take the time to really dig in. I know of no outlet that truly does that. It is almost as if a game is forgotten about after it is published and reviewed. The only post mortems out there of a game I know of are the ones that consist of an interview with a developer after the release or a write up the dev after release. While both enjoyable as a peek into the mind of the dev, they don't serve much of a journalistic or critical importance. Something I'm looking for more and more as I get older.

Another case in point but a bit off topic. Nothing to do with reviews but about depth. Geoff Keighley's Behind the Games series he did for Gamespot years back. I enjoyed those to no end and miss those very much. Maybe the tight PR machine that surrounds games now would never allow a reporter to get that deep into the workings of a developer but that is something I yearn for in games journalism today but haven't found since then.

cityshlacker said...

What a brilliant read. I've shown this to all of my friends that are involved in gaming one way or another, and they agreed that this is absolutely amazing. I don't have to contribute anything to this symposium myself though.

What struck me as odd was that I read everything in one session. I am not used to reading anything on my computer screen, but I was completely hooked.

I wish you the best of luck with your endeavours at 2K Boston.

Anonymous said...

電報opiopuy探偵entyouyouliaolaoren
qwowertofor名刺livehiskypercou名刺作成xieweishilookenw

Anonymous said...

govlog
桜の花
探偵
電報
名刺
オーク
フレッツ
腰痛
不動産
名刺作成

Anonymous said...

lorockmikrombardgovloggovlog 03170govlogoverAll ofraidagain
Fredericksodaswill be set freekeeping

Anonymous said...

Do you know Archlord gold? I like it.
My brother often go to the internet bar to buy Archlord money and play it.
After school, He likes playing games using these archlord online Gold with his friend.
I do not like to play it. Because I think that it not only costs much money but also spend much time. One day, he give me many cheap Archlord gold and play the game with me.
I came to the bar following him and found buy Archlord gold was so cheap. After that, I also go to play game with him.


Do you know cabal online alz? I like it.
My brother often go to the internet bar to buy cabal alz and play it.
After school, He likes playing games using these cabal gold with his friend.
I do not like to play it. Because I think that it not only costs much money but also spend much time. One day, he give me many cabal money and play the game with me.
I came to the bar following him and found buy cabal alz was so cheap. After that, I also go to play game with him.

Anonymous said...

Their studio is often overlooked, however, when discussing long lasting and well made games, to the great detriment of the MMO industry. With both games, it would be a true tragedy to ignore the story and fun that are available at a cost that is both reasonable and affordable compared to most other subscription MMOs. You will like the game and wakfu kama, I think.

More Console said...

Check out my interview with Leigh Alexander. Those of you with a sensitive gag reflex may wish to steer clear:

http://digitalgigolo.blogspot.com/2009/07/exclusive-leigh-alexander-interview.html

Anonymous said...

You are right, I believe that there will be many readers like you
Chanel Cambon?Christian Louboutin?Chanel Bags Chanel Clutches Jimmy Choo

Chanel Denim Handbags?Manolo Blahnik Chanel nappa handbags

Yves Saint Lauret?Christian Louboutin Boots Gucci Handbagsdiscount christianlouboutinManolo Blahnik Gucci Bags?Jimmy Choo? Gucci Wallets?Louis Vuitton BagsYves Saint Lauret? YSL?Christian

Louboutin
?http://www.bestlouisvuitton.com/http://www.buylouboutin.com/http://www.yitingbuy.com/http://www.discount-christianlouboutin.com/
 

Unknown said...

Remise Chaussures Sport
nike tn requin
chaussure nike tn requin

Unknown said...

do you like my products:
Air Max
wholesale polo shirts
Tn Requin
Ralph Lauren
Chaussures Nike
Wholesale Polo Shirts
cheap polo shirts
Polo Shirts
Air Shoes
Tennis Racquet Shop
Cheap Polo Shirts
The North Face Jackets
Cheap Nike Shoes
Nike Air Rift
chaussures puma
polo shirts
cheap nike shox
jeans diesel
Tn Requin
Tennis Racquet
ed hardy

kimberly sayer said...

Sean Michael Elliott (born February 2, 1968 in Tucson, Arizona) is a retired basketball player of NBA.Elliott played basketball at the Institute in Tucson Cholla, and the University of Arizona under the tutelage of Lute Olson.
sport fishing
fishing guide


http://www.fishingcostaricaexperts.com

Anonymous said...

Chaussures Sport
Chaussures Sport
tn requin
Air Jordan Shoes
NFL Jerseys

Anonymous said...

MBT shoes
nike tn
puma chaussures hommes
NFL Jerseys
Affliction Jeans
NFL Jerseys

macafe said...

Can't wait to see the fruits of your labors from 2K Boston as well.
That'Good Post.
China Business
Hong Kong Business
International Business
Business Opportunities
China Manufacturer
China Market
Picture
Computer
Computer Peripheral
Electronic
That a thank you

Unknown said...

designer sunglasses
Wholesale Polo Shirts
Chaussures Sport
MBT shoes
cheap cell phones
Cheap shoes online

Unknown said...

clothes jeans
Nike Sports Shoes
Nike Shox Shoes
Chaussures Sport
MBT shoes
Wholesale Polo Shirts
cheap cell phones
cheap shoes
polo shirts
nike tn
The North Face Jackets
Tennis Racquet
wholesale polo shirts
Air Jordan Shoes

Unknown said...

Look,Cheap GHD Straighteners,Discount GHD Styler,GHD Outlet,CHI hair Straighteners,2010 Cheap Sunglasses,Discount SunglassesSunglasses Outlet,wholesale Sunglasses,Sunglasses Wholesale. Do not worry. The urge to buy these goods will be very strong once you spot the shoes or clothes that you like. I am also a big fan for these products, although I like ED Hardy,cheap ED Hardy Clothes,discount ED Hardy Outlet,Wholesale Ed Hardy,Ed hardy wholesale, cheap Prada,discount Prada,Prada Outlet,newest Prada handbags,Prada handbags 2010, so I can understand. Online wholesale dealers have quite an array for you to choose from, including wholesale jewelry and handbags. And these are bags that bear the most famous names in the world of fashion, by the way -Prada, Spyder, Moncler, GHD, ED Hardy, designer Sunglasses, Ankh Royalty, Twisted Heart, or sayAnkhRoyalty,Ankh Royalty,Ankh Royalty Clothing ,Ankh Royalty Clothes,AnkhRoyalty Outlet. Such variety that comes at incredibly affordable prices is a treat you can't possibly pass up on.

Unknown said...

Look,Cheap GHD Straighteners,Discount GHD Styler,GHD Outlet,CHI hair Straighteners,2010 Cheap Sunglasses,Discount SunglassesSunglasses Outlet,wholesale Sunglasses,Sunglasses Wholesale. Do not worry. The urge to buy these goods will be very strong once you spot the shoes or clothes that you like. I am also a big fan for these products, although I like ED Hardy,cheap ED Hardy Clothes,discount ED Hardy Outlet,Wholesale Ed Hardy,Ed hardy wholesale, cheap Prada,discount Prada,Prada Outlet,newest Prada handbags,Prada handbags 2010, so I can understand. Online wholesale dealers have quite an array for you to choose from, including wholesale jewelry and handbags. And these are bags that bear the most famous names in the world of fashion, by the way -Prada, Spyder, Moncler, GHD, ED Hardy, designer Sunglasses, Ankh Royalty, Twisted Heart, or sayAnkhRoyalty,Ankh Royalty,Ankh Royalty Clothing ,Ankh Royalty Clothes,AnkhRoyalty Outlet. Such variety that comes at incredibly affordable prices is a treat you can't possibly pass up on.

Unknown said...

Look,Cheap GHD Straighteners,Discount GHD Styler,GHD Outlet,CHI hair Straighteners,2010 Cheap Sunglasses,Discount SunglassesSunglasses Outlet,wholesale Sunglasses,Sunglasses Wholesale. Do not worry. The urge to buy these goods will be very strong once you spot the shoes or clothes that you like. I am also a big fan for these products, although I like ED Hardy,cheap ED Hardy Clothes,discount ED Hardy Outlet,Wholesale Ed Hardy,Ed hardy wholesale, cheap Prada,discount Prada,Prada Outlet,newest Prada handbags,Prada handbags 2010, so I can understand. Online wholesale dealers have quite an array for you to choose from, including wholesale jewelry and handbags. And these are bags that bear the most famous names in the world of fashion, by the way -Prada, Spyder, Moncler, GHD, ED Hardy, designer Sunglasses, Ankh Royalty, Twisted Heart, or sayAnkhRoyalty,Ankh Royalty,Ankh Royalty Clothing ,Ankh Royalty Clothes,AnkhRoyalty Outlet. Such variety that comes at incredibly affordable prices is a treat you can't possibly pass up on.

Anonymous said...

I feel honored and privileged after reading your posts.You know,summer is coming! The homme chaussure will become fashionable soon.We offer hommes chaussures discount,all kinds,all colors available.You can also find hommes chaussures Nike,hommes chaussures pas cher.fashionable Chaussures Homme 2010,Best wishes to you!
Acheter hommes chaussures in our shop

Unknown said...

https://www.blogger.com/comment.g?blogID=10592227&postID=112144985420972854&page=1&token=1270620524559_AIe9_BHNiyUthsRPfHrFPpmsnmUp400GaDINhxy_L2whqAaTha8kq1SWbvvJbtyydNOBvxisJlWLF7T_lDPqUxvIufYSPzMy6Js7JseJ_PC-pwMEUG33vphDmpkeEoqyV2j2fSPkX-Kp7TuZDWwnaTUO0e6YIQVIkewz2gyk1gQVukpnMrkvHtIZaqyj_DyVf3G2AI9S5UxYyNWLTPHMWXO7EEcDkRdX0SA5cUdG-_LaHq-adjj8Niw

men's clothing said...

I feel honored and privileged after reading your posts.You know,summer is coming! chaussures homme online .The chaussures homme nike will become fashionable soon.We offer chaussures homme nike,all kinds,all colors available.You can also find homme chaussure in our shop,including hommes chaussures,Nike Shox NZ,Nike Shox R4.Best wishes to you!

chaussures mbt homme
chaussures homme nike
homme chaussure
nike air max 180
nike shox r6
nike shox r3
nike shox r4
nike shox r5
nike shox nz
chaussures femme nike

Cathy Mena said...

Since building my Asteroids arcade cabinet replica/ television cabinet, My videogame collection has grown almost double the size, I've completely filled the storage space in the cabinet and now there are boxes, consoles and loose carts all over the floor.
wholesale caps

men's clothing said...

I feel honored and privileged after reading your posts.You know,summer is coming! chaussures homme online .The chaussures homme

nike will become fashionable soon.We offer chaussures homme nike,all kinds,all colors

available.You can also find homme chaussure

in our shop,including hommes chaussures,Nike

Shox NZ,Nike Shox R4.Best wishes to you!
chaussures mbt

homme

chaussures mbt

femme

chaussures

homme pas cher


homme chaussure
nike air max 180
nike air max 95
nike air max 90
nike shox r6
nike shox r3
nike shox r4
nike shox r5
nike shox nz
nike tn dollar
nike tn spider
chaussures femme

nike

nike dunk sb

low

Christian Hymon said...

thanks for the post!!!

Air max chaussures : http://www.air-max-chaussures.com/
Air max 1 chaussures : http://www.air-max-chaussures.com/air-max-1-c-86.html
Air max 90 chaussures: http://www.air-max-chaussures.com/air-max-90-c-89.html
Air max light chaussures : http://www.air-max-chaussures.com/air-max-light-c-94.html
Air pegasus 89 chaussures: http://www.air-max-chaussures.com/air-pegasus-89-c-95.html
Air 180 chaussures: http://www.air-max-chaussures.com/air-180-chaussures-c-96.html
Air structure triax 91 chaussures:
http://www.air-max-chaussures.com/air-structure-triax-91-c-91.html
Air classic BW chaussures: http://www.air-max-chaussures.com/air-classic-bw-c-97.html
Air Tailwind 92 chaussures:
http://www.air-max-chaussures.com/air-tailwind-92-c-93.html
Air max 93 chaussures
http://www.air-max-chaussures.com/air-max-93-c-85.html
Air max 95 chaussures
http://www.air-max-chaussures.com/air-max-95-c-99.html
Air max 97 chaussures
http://www.air-max-chaussures.com/air-max-97-c-101.html
Air max tn chaussures
http://www.air-max-chaussures.com/air-max-tn-c-103.html
Air max ltd chaussures
http://www.air-max-chaussures.com/air-max-ltd-c-108.html
Air max 360 chaussures
http://www.air-max-chaussures.com/air-max-360-c-112.html
nice work.

ClothingJeans said...

We provide flare True Religion Jeans of very low Price. And also Wholesale Evisu JeansYou can choose the cheap True Religion Jeans from our online shop.

Unknown said...

chaussures mbt homme
chaussures mbt femme
chaussures homme nike

ai said...

ugg boots
polo boots
polo shoes

herve leger
herve leger bandage dress

chanel outlet
chanel bandbags
chanel bags
chanel iman

ralph Lauren polo
ralph lauren outlet
lacoste polo
polo raplh lauren

air jordan 2010
cheap jordan shoes
jordan ajf shoes
discount jordan shoes
jumpman23

moncler
moncler jackets
moncler coats
moncler vest
moncler outlet
moncler Polo t shirt
cheap five finger shoes

kiss ghd

Unknown said...

My favorite brand is ed hardy wholesale, and I always want this brand of
wholesale handbags. And its T-shirts are very special too. I really hope I can wear its
wholesale t-shirtsto go to school and go shopping with my friends, and it will be awesome. And most of my friends like the Ed Hardy style too.
However, this brand is too expensive for me to buy. An Ed Hardy Nike wholesale cost nearly 250 dollars. It is almost a cost of my living of mouth. I am only an ordinary student who has no trust fund; no job and no income that I can’t afford to pay such high price for its sunglasses wholesaleor wholesale ghd. So, I won’t buy the Ed Hardy brand series at least for now.

Unknown said...

buy Casodex

ahmet said...

Interesting Article. Hoping that you will continue posting an article having a useful information. Thanks a lot! | sınırsız porno izle | porno izle | gizli çekim sikiş | porno izle | porno izle | bakire | adult hikaye | kerhane | sikiş sahneleri | am yalama | tecavüz | oyun oyna | süper oyunlar | göt sikme | kara şimşek | orgazm | jenna jameson | gabriella fox | bakire sikiş | annina ucatis | genelevde sikiş | sınırsız porno izle | sikişme sahneleri | ev pornosu | panda movie es | süper sikiş | sikişen karılar | lezbiyen am yalama | sikme | yarak yalama | rokettube | sikişme | evli türk pornosu | tecavüz | ama sokuş |

ahmet said...

| pornoxxx | japon sikişi | sarışın sikiş | azgın sikiş | erotikhikayeoku | tecavüz hikayeleri | sikiş hikayeleri | sadie west | tatlı sikiş | sulu amcık | sinirsiz porno | azgın sikiş | transeksüel sikiş | adult hikaye | fetis hikayeler | hikaye | anal hikayeler | anal sikiş | bağırtarak sikiş | götten sikiş | köpek stilinde sikiş | karısını sikiyor | tatlı amcık | tatlı sikiş | partide sikiş | porno yıldızı | sikişme | sikiş izle | amcık sex izle | türk pornosu sikiş | travesti pornosu | türk porno güzelleri | olgun kadınların sikiş filmi | becermek | ama boşalmak | 31 çekiş | am sikme | domaltarak sikiş | kızlık bozma | sexmekan | gerdek gecesi | emenuele | borno

Unknown said...

NewStreetFashion
Ed Hardy
stylish design
Ed Hardy Wholesale
fashion excellent quality
wholesale Ed Hardy
ED Hardy clothing bring you a super surprise!
ed hardy wholesale clothing
The quality is so good
christian audigier

Unknown said...

My father recommended your post. I've bookmarked it.
china wholesale
wholesale golf clubs
wholesale handbags
wholesale cell phones
wholesale power balance
wholesale ipod
wholesale iphone
wholesale memory cards
wholesale sporting goods
wholesale Wedding dresses
wholesale jewelry
Most Salable Golf Products at Unparalleled Price
Summer Cool Wear and Decoration Accessories
power balance
2010 FIFA World Cup South Africa

Anonymous said...

abercrombie and

fitch

abercrombie &

fitch

Abercrombie and fitch outlet
nike shoes

wholesale

wholesale

clothing

china wholesale

china wholesale for linkchina said...

Nice send! GA is also my biggest earning. However, it’s not just a very much. many thanks !! incredibly beneficial ...
china wholesale
wholesale golf clubs
wholesale handbags
wholesale cell phones
wholesale power balance
wholesale ipod accessories
wholesale memory cards
wholesale sporting goods
wholesale Wedding dresses
wholesale jewelry
Most Salable Golf Products at Unparalleled Price
Summer Cool Wear and Decoration Accessories
2010 FIFA World Cup South Africa

Unknown said...

Interesting Article. Hoping that you will continue posting an article having a useful information. Thanks a lot! | sınırsız porno izle | sikiş videoları | porno izle | gizli çekim sikiş | porno izle | porno izle | bakire | adult hikaye | kerhane | sikiş sahneleri | am yalama | tecavüz | oyun oyna | süper oyunlar | göt sikme | kara şimşek | orgazm | jenna jameson | gabriella fox | bakire sikiş | annina ucatis | genelevde sikiş | sınırsız porno izle | sikişme sahneleri | ev pornosu | panda movie es | süper sikiş | sikişen karılar | lezbiyen am yalama | sikme | yarak yalama | rokettube | sikişme | evli türk pornosu | tecavüz | ama sokuş | pornoxxx | japon sikişi | sarışın sikiş | azgın sikiş | erotikhikayeoku | tecavüz hikayeleri | sikiş hikayeleri | sadie west | tatlı sikiş | sulu amcık | sinirsiz porno | azgın sikiş | transeksüel sikiş | adult hikaye | fetis hikayeler | hikaye

Unknown said...

transeksüel sikiş | adult hikaye | fetis hikayeler | hikaye | anal hikayeler | anal sikiş | bağırtarak sikiş | götten sikiş | köpek stilinde sikiş | karısını sikiyor | tatlı amcık | tatlı sikiş | partide sikiş | porno yıldızı | sikişme | sikiş izle | amcık sex izle | türk pornosu sikiş | travesti pornosu | türk porno güzelleri | olgun kadınların sikiş filmi | becermek | ama boşalmak | 31 çekiş | am sikme | domaltarak sikiş | kızlık bozma | sexmekan | gerdek gecesi | emenuele | borno | porno videoları | sevişme videoları | liseli sikiş videoları | liseli sikiş izle | hamile sikiş | ateşli porno | erotik film izle | erotik sikiş | ünlülerin pornoları | işemeli porno | işemeli sikiş

combattery84 said...

Battery
Laptop Battery
Camcorder Battery
Digital Camera Battery
Mobile Phone Battery
PDA Battery
acer laptop battery
asus laptop battery
apple laptop battery
dell laptop battery
fujitsu laptop battery
hp laptop battery
ibm laptop battery
sony laptop battery
toshiba laptop battery
CANON Camcorder Battery
JVC Camcorder Battery
PANASONIC Camcorder Battery
SHARP Camcorder Battery
SONY Camcorder Battery
NOKIA Mobile Phone Battery
Laptop Battery
acer Laptop Battery
apple Laptop Battery
asus Laptop Battery
compaq Laptop Battery
Dell Laptop Battery
fujitsu Laptop Battery
gateway Laptop Battery
hp Laptop Battery
ibm Laptop Battery
sony Laptop Battery
toshiba Laptop Battery

combattery84 said...

SHARP Camcorder Battery
CANON Digital Camera Battery
CASIO Digital Camera Battery
FUJIFILM Digital Camera Battery
JVC Digital Camera Battery
NIKON Digital Camera Battery
SANYO Digital Camera Battery
SHARP Digital Camera Battery
SONY Digital Camera Battery
PANASONIC Digital Camera Battery
OLYMPUS Digital Camera Battery
IBM Laptop Battery
acer laptop battery
apple Laptop Battery
toshiba laptop battery
hp latptop battery
dell laptop battery
sony laptop battery
asus laptop battery
compaq laptop battery
FUJITSU Laptop Battery

LG Laptop Battery
Laptop Battery
ACER Laptop Battery
ASUS Laptop Battery
COMPAQ Laptop Battery
Dell Laptop Battery
FUJITSU Laptop Battery
HP Laptop Battery
IBM Laptop Battery

combattery84 said...

SAMSUNG Laptop Battery
SONY Laptop Battery
TOSHIBA Laptop Battery
Battery
Laptop Battery
Camcorder Battery
Digital Camera Battery
Mobile Phone Battery
PDA Battery
acer laptop battery
asus laptop battery
apple laptop battery
dell laptop battery
fujitsu laptop battery
hp laptop battery
ibm laptop battery
sony laptop battery
toshiba laptop battery
CANON Camcorder Battery
JVC Camcorder Battery
PANASONIC Camcorder Battery
SHARP Camcorder Battery
SONY Camcorder Battery
NOKIA Mobile Phone Battery
Laptop Battery
acer Laptop Battery
apple Laptop Battery
asus Laptop Battery
compaq Laptop Battery
Dell Laptop Battery
fujitsu Laptop Battery
gateway Laptop Battery
hp Laptop Battery
ibm Laptop Battery
sony Laptop Battery
toshiba Laptop Battery

combattery84 said...

SONY Camcorder Battery
SHARP Camcorder Battery
CANON Digital Camera Battery
CASIO Digital Camera Battery
FUJIFILM Digital Camera Battery
JVC Digital Camera Battery
NIKON Digital Camera Battery
SANYO Digital Camera Battery
SHARP Digital Camera Battery
SONY Digital Camera Battery
PANASONIC Digital Camera Battery
OLYMPUS Digital Camera Battery
IBM Laptop Battery
acer laptop battery
apple Laptop Battery
toshiba laptop battery
hp latptop battery
dell laptop battery
sony laptop battery
asus laptop battery
compaq laptop battery
FUJITSU Laptop Battery

LG Laptop Battery
Laptop Battery
ACER Laptop Battery
ASUS Laptop Battery
COMPAQ Laptop Battery
Dell Laptop Battery
FUJITSU Laptop Battery

combattery84 said...

SONY Camcorder Battery
SHARP Camcorder Battery
CANON Digital Camera Battery
CASIO Digital Camera Battery
FUJIFILM Digital Camera Battery
JVC Digital Camera Battery
NIKON Digital Camera Battery
SANYO Digital Camera Battery
SHARP Digital Camera Battery
SONY Digital Camera Battery
PANASONIC Digital Camera Battery
OLYMPUS Digital Camera Battery
IBM Laptop Battery
acer laptop battery
apple Laptop Battery
toshiba laptop battery
hp latptop battery
dell laptop battery
sony laptop battery
asus laptop battery
compaq laptop battery
FUJITSU Laptop Battery

LG Laptop Battery
Laptop Battery
ACER Laptop Battery
ASUS Laptop Battery
COMPAQ Laptop Battery
Dell Laptop Battery
FUJITSU Laptop Battery

combattery84 said...

dell laptop battery
fujitsu laptop battery
hp laptop battery
ibm laptop battery
sony laptop battery
toshiba laptop battery
CANON Camcorder Battery
JVC Camcorder Battery
PANASONIC Camcorder Battery
SHARP Camcorder Battery
SONY Camcorder Battery
NOKIA Mobile Phone Battery
Laptop Battery
acer Laptop Battery
apple Laptop Battery
asus Laptop Battery
compaq Laptop Battery
Dell Laptop Battery
fujitsu Laptop Battery
gateway Laptop Battery
hp Laptop Battery
ibm Laptop Battery
sony Laptop Battery

combattery84 said...

HP Laptop Battery
IBM Laptop Battery
SONY Laptop Battery
TOSHIBA Laptop Battery
Acer btp-arj1 battery
laptop battery
ACER Laptop Battery
APPLE Laptop Battery
ASUS Laptop Battery
COMPAQ Laptop Battery
Dell Laptop Battery
FUJITSU Laptop Battery
GATEWAY Laptop Battery
HP Laptop Battery
IBM Laptop Battery
LG Laptop Battery
SAMSUNG Laptop Battery
SONY Laptop Battery
TOSHIBA Laptop Battery
Battery
Laptop Battery
Camcorder Battery
Digital Camera Battery
Mobile Phone Battery
PDA Battery
acer laptop battery
asus laptop battery
apple laptop battery

combattery84 said...

ACER aspire 5560 battery
ACER BATBL50L6 battery
ACER TravelMate 240 Battery
ACER BT.00803.004 Battery
ACER Travelmate 4002lmi battery
Acer travelmate 800 battery
Acer aspire 3613wlmi battery
Travelmate 2414wlmi battery
Acer batcl50l battery
Acer Travelmate 2300 battery
ACER aspire 3610 battery
ACER travelmate 4600 battery
Dell Latitude D800 battery
Dell Inspiron 600m battery
Dell Inspiron 8100 Battery
Dell Y9943 battery
Dell Inspiron 1521 battery
Dell Inspiron 510m battery
Dell Latitude D500 battery
Dell Latitude D520 battery
Dell GD761 battery
Dell NF343 battery
Dell D5318 battery
Dell G5260 battery

combattery84 said...

Dell Inspiron 9200 battery
Dell Latitude C500 battery
Dell HD438 Battery
Dell GK479 battery
Dell PC764 battery
Dell KD476 Battery
Dell Inspiron 1150 battery
Dell inspiron 8500 battery
Dell Inspiron 4100 battery
Dell Inspiron 4000 battery
Dell Inspiron 8200 battery
Dell FK890 battery
Dell Inspiron 1721 battery
Dell Inspiron 1300 Battery
Dell Inspiron 1520 Battery
Dell Latitude D600 Battery
Dell XPS M1330 battery
Dell Latitude D531N Battery
Dell INSPIRON 6000 battery
Dell INSPIRON 6400 Battery
Dell Inspiron 9300 battery
Dell INSPIRON 9400 Battery
Dell INSPIRON e1505 battery
Dell INSPIRON 2500 battery
Dell INSPIRON 630m battery

combattery84 said...

Dell Latitude D820 battery
Dell Latitude D610 Battery
Dell Latitude D620 battery
Dell Latitude D630 battery
Dell xps m1210 battery
Dell e1705 battery
Dell d830 battery
Dell inspiron 2200 battery
Dell inspiron 640m battery
Dell inspiron b120 battery
Dell xps m1210 battery
Dell inspiron xps m1710 battery
Dell inspiron 1100 battery

Dell 310-6321 battery
Dell 1691p battery
Dell Inspiron 500m battery
Dell 6Y270 battery
Dell inspiron 8600 battery
Latitude x300 series battery
Dell latitude cpi battery
Dell 1x793 battery
dell Inspiron 1501 battery
Dell 75UYF Battery
Dell Inspiron 1720 battery

Unknown said...

Christian Louboutin Pumps
michael jordan shoes
Mens Polo Shirts
Mlb Jerseys

Manolo blahnik shoes
jordan basketball shoes
Lacoste Polo Shirts
Nba jerseys

Christian Louboutin Shoes Sale
jordan shoes sale
Wholesale Polo Shirts
Authentic Nhl jerseys
kids Classic Ugg Boots
Metallic Uggs
Balmain Shoes
Metallic Uggs
Mini Ugg Boots
Alexander McQueen
Mini Ugg Boots
Nightfall Uggs
Chanel Shoes
Nightfall Uggs
Stripe Cable Ugg Boots
Gucci Shoes
Stripe Cable Ugg Boots
Sundance Uggs
Tory Burch Shoes
Sundance Uggs
Tall Romantic Flower Uggs
Bottega Veneta
Tall Romantic Flower Uggs
UGG Classic Tall Boots

trassy dicosta said...

Send flowers to poland

I recently came across your blog and have been reading along. I think I will leave my first comment. I don’t know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often.

UAE Flowers Shop said...

Nice effort, very informative, this will help me to complete my task..I love flowers...I am also being a part of flowers delivery all over the world.

UAE Flowers Shop said...

Nice effort, very informative, this will help me to complete my task..I love flowers...I am also being a part of flowers delivery all over the world.

Anonymous said...

Acer Aspire 1410 battery
Acer ASPIRE 1680 battery
ACER BTP-63D1 battery
ACER BTP-43D1 battery
Acer lc.btp05.001 battery
Acer aspire 3000 battery
Acer Travelmate 4000 battery
ACER aspire 5560 battery
ACER BATBL50L6 battery
ACER TravelMate 240 Battery
ACER BT.00803.004 Battery
ACER Travelmate 4002lmi battery
Acer travelmate 800 battery
Acer aspire 3613wlmi battery
Travelmate 2414wlmi battery
Acer batcl50l battery
Acer Travelmate 2300 battery
ACER aspire 3610 battery
ACER travelmate 4600 battery
Dell Latitude D800 battery
Dell Inspiron 600m battery
Dell Inspiron 8100 Battery
Dell Y9943 battery
Dell Inspiron 1521 battery
Dell Inspiron 510m battery
Dell Latitude D500 battery
Dell Latitude D520 battery
Dell GD761 battery
Dell NF343 battery
Dell D5318 battery

Anonymous said...

Dell Latitude D620 battery
Dell Latitude D630 battery
Dell xps m1210 battery
Dell e1705 battery
Dell d830 battery
Dell inspiron 2200 battery
Dell inspiron 640m battery
Dell inspiron b120 battery
Dell xps m1210 battery
Dell inspiron xps m1710 battery
Dell inspiron 1100 battery
Dell 310-6321 battery
Dell 1691p battery
Dell Inspiron 500m battery
Dell 6Y270 battery
Dell inspiron 8600 battery
Latitude x300 series battery
Dell latitude cpi battery
Dell 1x793 battery
dell Inspiron 1501 battery
Dell 75UYF Battery
Dell Inspiron 1720 battery
dell Latitude C640 battery
Dell XPS M140 battery
Dell Inspiron E1405 battery
dell 700m battery
dell C1295 battery
Dell U4873 Battery
Dell Latitude C600 battery
Armada E700 Series battery
Compaq 116314-001 battery
Compaq 319411-001 battery

Anonymous said...

Compaq nc4200 battery
Compaq Presario R3000 Battery
Compaq Presario 2100 battery
Compaq Presario r3000 Battery
Compaq Business Notebook NX9000 battery
HP 395789-001 battery
HP 446506-001 Battery
HP dv9700 battery
HP F4809A Battery
HP nc8000 battery
HP nc8230 battery
HP pavilion zd8000 battery
HP f2024b battery
HP f4812a battery
HP Pavilion ZV5000 battery
HP Pavilion DV1000 battery
HP Pavilion ZD7000 Battery
HP Pavilion DV2000 battery
HP Pavilion DV4000 Battery
HP Pavilion dv6000 Battery
HP Pavilion DV9000 Battery
HP F4098A battery
HP pavilion zx6000 battery
HP omnibook xe4400 battery
HP omnibook xe4500 battery
HP omnibook xe3 battery
Notebook NX9110 battery
IBM 02K6821 battery
IBM 02K7054 battery
IBM 08K8195 battery

Anonymous said...

FUJITSU Fpcbp79 Battery
FUJITSU Fpcbp95 Battery
FUJITSU Fpcbp98 Battery
FUJITSU Fpcbp121 Battery
FUJITSU Fpcbp151 Battery
FUJITSU lifebook t4010 Battery
FUJITSU lifebook t4020d Battery
GATEWAY NX7000 battery
UNIWILL 258-4S4400-S1P1 Battery
TOSHIBA PA3307U-1BRS Battery
TOSHIBA PA3383U-1BRS Battery
TOSHIBA PA3384U-1BRS Battery
TOSHIBA PA3465U-1BRS Battery
Toshiba PA2487UR battery
Toshiba A100 Battery
Toshiba Satellite A105 battery
Toshiba A70 battery
PA3062U-1BAT battery
Toshiba Satellite P30 battery
Toshiba PA3084U-1BRS battery
Toshiba PA3098U battery
PA3107U-1BAS battery
PA3107U-1BRS battery
PA3166U-1BRS battery
PA3176U-1BAS battery
TOSHIBA PABAS076 Battery

Anonymous said...

Dell Inspiron 510m battery
Dell Latitude D500 battery
Dell Latitude D520 battery
Dell GD761 battery
Dell NF343 battery
Dell D5318 battery
Dell G5260 battery
Dell Inspiron 9200 battery
Dell Latitude C500 battery
Dell HD438 Battery
Dell GK479 battery
Dell PC764 battery
Dell KD476 Battery
Dell Inspiron 1150 battery
Dell inspiron 8500 battery
Dell Inspiron 4100 battery
Dell Inspiron 4000 battery
Dell Inspiron 8200 battery
Dell FK890 battery
Dell Inspiron 1721 battery
Dell Inspiron 1300 Battery
Dell Inspiron 1520 Battery


Dell Latitude D600 Battery
Dell XPS M1330 battery
DELL XPS M1730 Battery
Dell Latitude D531N Battery
Dell INSPIRON 6000 battery
Dell INSPIRON 6400 Battery
Dell Inspiron 9300 battery
Dell INSPIRON 9400 Battery
Dell INSPIRON e1505 battery

Anonymous said...

Dell INSPIRON 2500 battery
Dell INSPIRON 630m battery
Dell Latitude D820 battery
Dell Latitude D610 Battery
Dell Latitude D620 battery
Dell Latitude D630 battery
Dell xps m1210 battery
Dell e1705 battery


Dell d830 battery
Dell inspiron 2200 battery
Dell inspiron 640m battery
Dell inspiron b120 battery
Dell xps m1210 battery
Dell inspiron xps m1710 battery
Dell inspiron 1100 battery
Dell 310-6321 battery
Dell 1691p battery
Dell Inspiron 500m battery
Dell 6Y270 battery
Dell inspiron 8600 battery
Latitude x300 series battery
Dell latitude cpi battery
Dell 1x793 battery
dell Inspiron 1501 battery
Dell 75UYF Battery
Dell Inspiron 1720 battery
dell Latitude C640 battery
Dell XPS M140 battery
Dell Inspiron E1405 battery

Anonymous said...

Dell TK330 Battery
Dell M1530 Battery
Dell Latitude C600 battery
Armada E700 Series battery
Compaq 116314-001 battery
Compaq 319411-001 battery
Compaq nc4200 battery
Compaq Presario R3000 Battery
Compaq Presario 2100 battery
Compaq Presario r3000 Battery
Compaq Business Notebook NX9000 series battery
HP 395789-001 battery
HP 446506-001 Battery
HP dv9700 battery
HP F4809A Battery
HP nc8000 battery
HP nc8230 battery
HP pavilion zd8000 battery
HP f2024b battery
HP f4812a battery
HP Pavilion ZV5000 battery
HP Pavilion DV1000 battery
HP Pavilion ZD7000 Battery
HP Pavilion DV2000 battery
HP Pavilion DV4000 Battery
HP Pavilion dv6000 Battery
HP Pavilion DV9000 Battery
HP F4098A battery

Anonymous said...

IBM thinkpad t42 battery
IBM thinkpad r51 battery
Thinkpad r50 series battery
IBM thinkpad r32 battery
Thinkpad x41 series battery
SONY VGP-BPS2 Battery
SONY VGP-BPS2C Battery
SONY VGP-BPS5 battery
SONY VGP-BPL2C battery
SONY VGP-BPS2A battery
SONY VGP-BPS2B battery
SONY PCGA-BP1N battery
SONY PCGA-BP2E battery
SONY PCGA-BP2NX battery
SONY PCGA-BP2S battery
SONY PCGA-BP2SA battery
SONY PCGA-BP2T battery
SONY PCGA-BP2V battery
SONY PCGA-BP4V battery
SONY PCGA-BP71 battery
SONY PCGA-BP71A battery
SONY VGP-BPL1 battery
SONY VGP-BPL2 battery
Sony vgn-t2xp/s battery
Sony vaio vgn-s4xp battery
Sony vaio pcg-z1rsp battery
SONY NP-FT1 battery
SONY NP-FC10 Battery

Vidmar said...

Home Wind Power
Child Custody Rights
Weber Grill 22.5
Beat Making Programs
Brother Embroidery Machines
Cheapest Gold

tn requin said...

rosetta stone
rosetta stone
rosetta stone spanish
rosetta stone 3.4.5
rosetta stone
rosetta stone spanish
rosetta stone 3.4.5
rosetta stone
rosetta stone spanish
rosetta stone software
rosetta stone spanish
rosetta stone software
rosetta stone
rosetta stone spanish
rosetta stone 3.4.5

jack chan said...

65068094lv cosmic blossom
cosmic blossom
louis vuitton cosmic blossom
lv store
lv purses
Soft Briefcase
batignolles horizontal louis vuitton
galliera lv
louis vuitton damier azur handbag
louis vuitton damier speedy 30
louis vuitton florin
louis vuitton damier neverfull mm
bulles
louis vuitton vavin
louis vuitton zippy wallet
louis vuitton laptop
laptop sleeve 13
louis vuitton boetie gm
louis vuitton epi leather
epi leather
louis vuitton anton
Sac Bosphore

Unknown said...

grow taller Electronic Cigarette reverse phone detective tava tea error fix

jimmychooshoes said...

|Jimmy Choo Private strappy sandals
|Jimmy Choo Reveal jeweled suede sandals
|Jimmy Choo Ryder Embossed Leather Sandals Pump
|Jimmy Choo Shaded Elaphe snakeskin sandals
|Jimmy Choo Silk satin peep toe sandals
|Jimmy Choo Silver Patent Leather High-heeled Sandals
|Jimmy Choo Snake print leather sandals
Christian Louboutin Tell Me Peep-Toe Pumps
Christian Louboutin Very Prive Peep-Toe Pumps Fuschia sequins
christian louboutin Alta Bouton Ankle Boots

weddingdresses said...

I just have this energy level that I've never had before. The only thing that's changed is I've started wearing my Power Balance and nike running shoes
Power Balance wholesale

jimmychooshoes said...

Christian Louboutin Fontanete shoes platform pump |Christian Louboutin Fontanete shoes platform pump
Christian Louboutin Fontanete pink suede platform shoes |Christian Louboutin Fontanete pink suede platform shoes
Christian Louboutin Crystal Leather Gold Pumps |Christian Louboutin Crystal Leather Gold Pumps
Christian Louboutin Blue Satin Anemone Pumps |Christian Louboutin Blue Satin Anemone Pumps

Anonymous said...

Toysgiftsrus.com is a reputable company that stand out in customer services. You will have a piece of mind because not only they will make sure you get the Pillow Pets Toys and Games on time but they are very actively in updating you the status of your delivery. Gift wrapping is also available for just a small additional fee.

combattery84 said...

BOSCH power tool Battery
Dewalt power tool Battery
Makita power tool Battery
Panasonic power tool Battery
ACER AS07B42 Battery | ACER GRAPE34 Battery | ACER LC.BTP00.006 Battery | APPLE A1175 battery | APPLE A1189 battery | ASUS A32-F5 Battery | HP Pavilion dv2000 battery | HP Pavilion dv1000 battery | NX6120 battery | HP ZV5000 battery | hp 510 battery | HP F4809A battery | TOSHIBA PA3450U-1BRS battery | TOSHIBA PA3399U-1BRS battery | TOSHIBA PA3399U-1BAS battery | TOSHIBA PA3399U-2BRS battery | TOSHIBA PA3534U-1BRS battery | TOSHIBA PA3534U-1BAS battery | TOSHIBA PA3535U-1BAS battery | TOSHIBA PA3285U-3BRS battery | TOSHIBA PA3451U-1BRS battery | TOSHIBA PA3465U-1BRS battery | TOSHIBA PA3450U-1BRS battery | SONY PCGA-BP2NX battery | Dell inspiron 1420 battery | Dell m1330 battery | <a

Unknown said...

Can't wait to see the fruits of your labors from 2K Boston as well.

download youtube

徵信情感小屋 said...

Bless you Shawn. come on.








---------------------------
From to: teflon ptfe properties,Teflon PTFE manufacturer

Unknown said...

There's a line, I think, between making a prejudgment, and bringing with puma soldes you a context within which to make an evaluation.

boweiz said...

Look up tautology in the dictionary. It does not mean what pas cher timberland you think it means.

Sam said...

Well done. I m certainly floored with the excellence of the data presented. I hope that you keep up with the brilliant work succesfully done.
locksmith miami fl
Nashville Locksmith
Union City locksmith
Newark locksmith
Locksmith Newark CA
Newark locksmith
Locksmith Newark CA
Locksmith Newark CA
Elmhurst Locksmith
Locksmith Elmhurst
Locksmith Elmhurst
Locksmith Elmhurst IL
Locksmith Elmhurst IL
Elmhurst Locksmith
Locksmith Elmhurst
Locksmith Elmhurst
Locksmith Elmhurst IL
Locksmith Elmhurst IL
Locksmith Elmhurst
Locksmith Elmhurst
Elmhurst Locksmith
Elmhurst Locksmith
locksmith aventura
locksmith fort worth
locksmith fort worth
locksmith fort worth
locksmith aventura
locksmith fort worth
locksmith fort worth
locksmith aventura

sunnyakc said...

Elliott last summer when they decided to expand their conversation to a number of noted reviewers

Lauraine said...

Thanks for the post. It was very interesting and meaningful.
Angry birds clone

Lauraine said...

Thanks for the post. I liked it. Keep going I follow you.
Fiverr clone script

su422503 said...

アユーラ
グラスオール
マダムアルバ
化粧品
プロアクティブ
プロアクティブ
プロアクティブ
マダムアルバ
アテニア
ヒフミド
アグレーラ
草花木果
江原道 こうげんどう
アテニア
アスタリフト
草花木果

エニシングホワイト 天真堂
オラクル
グラスオール
プロアクティブ
スキンケア ランキング
基礎化粧品
プロアクティブ
なちゅライフ
アイムピンチ
マキアレイベル
ニュープロアクティブ
マリアンナプラス 豊麗
インナーシグナル
ヒフミドバッグ
お米でつるつる
お米でつるつる

su422503 said...

アユーラ
グラスオール
マダムアルバ
化粧品
プロアクティブ
プロアクティブ
プロアクティブ
マダムアルバ
アテニア
ヒフミド
アグレーラ
草花木果
江原道 こうげんどう
アテニア
アスタリフト
草花木果

エニシングホワイト 天真堂
オラクル
グラスオール
プロアクティブ
スキンケア ランキング
基礎化粧品
プロアクティブ
なちゅライフ
アイムピンチ
マキアレイベル
ニュープロアクティブ
マリアンナプラス 豊麗
インナーシグナル
ヒフミドバッグ
お米でつるつる
お米でつるつる

yyyy said...

Thanks for sharing and letting us aware about this information. This is a great thread, so much info.
http://www.2joygame.com/WOW-News/Blizzard-said-there-are-another-two-pieces-of-information-about-World-of-Warcraft.html

http://www.wowbotsfree.com/pros-and-cons-of-switching-between-warcraft-union-camps/

http://teralife.eklablog.com/sen-richard-blumenthal-s-d-conn-proposed-personal-data-protection-a5498541

Unknown said...

fashion jewelry wholesale
wholesale fashion jewelry
china jewelry wholesale
china fashion jewelry
jewelry wholesale china
fashion jewelry china
wholesale jewelry
jewelry wholesale
[URL=http://www.wonderful-jewelry.com]Fashion Jewelry[/URL],
[URL=http://www.wonderful-jewelry.com]fashion jewelry wholesale[/URL],
[URL=http://www.wonderful-jewelry.com]Wholesale Fashion Jewelry[/URL],
[URL=http://www.wonderful-jewelry.com]China Jewelry Wholesale[/URL],
[URL=http://www.wonderful-jewelry.com]China Fashion Jewelry[/URL]
[URL=http://www.wonderful-jewelry.com]Jewelry Wholesale China[/URL]
[URL=http://www.wonderful-jewelry.com]Fashion Jewelry China[/URL]

Henry County georgia said...

Thanks for the write up! Also, just a heads up, your RSS feeds aren't working. Could you take a look at that?

henry county ga tourist information

Henry County georgia said...

Thanks for the write up! Also, just a heads up, your RSS feeds aren't working. Could you take a look at that?

henry county ga tourist information

Anonymous said...

Thank you very much for this post, it is highly informative for artists who are considering doing book covers



Hotel Aguas De Ibiza

nashvile said...

I would like to thank you for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well.


Nashville Air Conditioning

san said...

Details are very true and factual, I do really learned something new upon reading this

interesting post. Good job! and Thanks!


San Diego Property Management

Anonymous said...

href="http://www.hightideprop.com/">San Diego Property Management"


San Diego Property Management Company. Hightideprop San Diego Property Manager provides professional

baby marino said...

href="http://www.nuiorganics.com /"> Baby Merino nz "


the native language of New Zealand, 'nui' means 'big' or 'important' hence the name Nui Organics™. Because we believe a clean start is a big thing for a little person.

Costa Rica Travel and Fishing said...

Costa Rica Fishing
A pretty hefty list to say the least. Along with these topics is a long list of major game industry journalists taking part including Dan Hsu, N’Gai Croal, Leigh Alexander, John Davison, Jeff Gerstmann, Stephen Totilo, and others. Costa Rica Fishing and Tours Each person has a unique perspective to bring into the conversation with Shawn playing moderator.

http://costa-ricafishing.com/
The first part of the symposium is up right now with the topic of review scores. The conversation bounces around from merits of reviews, to practices of reviews, to impact of reviews on the reviewer. Costa Rica Wedding It’s a fairly long read, but I think it’s important that most interested in reviews or games journalism do so. At a time when game magazines are dieing off, large games journalism sites are dwindling, and smaller game focused sites are rising I think these conversations should be had.

Costa Rica Fishing
It’ll be interesting to see the coming parts of the symposium as topics get more complex and gain relevance with recent events.
Costa Rica Fishing and Tours
http://costa-ricafishing.com/
Costa Rica Wedding
Game Reviews Symposium part 1

prodigyceo said...

I entered at the south end and rode north.


Ankara escort

Escort ankara

escort bayan ankara

Ankara escort

Escort ankara

escort bayan ankara

Ankara escort

Escort ankara

escort bayan ankara

escort bayan ankara kızılay

Escort ankara bayan

escort bayan ankara çankaya

Ankara escort bayan

Escort ankara bayan

escort bayan ankara

Ankara bayan escort

bayan escort ankara

escort bayan ankara

Ankara escort bayan

Escort ankara bayan

escort bayan ankara
Ankara escort bayan

Escort ankara bayan

escort bayan ankara

prodigyceo said...

I entered at the south end and rode north.


Ankara escort

Escort ankara

escort bayan ankara

Ankara escort

Escort ankara

escort bayan ankara

Ankara escort

Escort ankara

escort bayan ankara

escort bayan ankara kızılay

Escort ankara bayan

escort bayan ankara çankaya

Ankara escort bayan

Escort ankara bayan

escort bayan ankara

Ankara bayan escort

bayan escort ankara

escort bayan ankara

Ankara escort bayan

Escort ankara bayan

escort bayan ankara
Ankara escort bayan

Escort ankara bayan

escort bayan ankara

ema said...

Any sacks were definitely a good adequately inviting solution - beats headphones with grained synthetic leather weekenders together with wash cloth monogram gym floor sacks that will graphic ebony checkered luggage together with russet hued tote handbags the fact that glistened such as the diligently elderly base. Truly, if ever the clearly show have all super star it had become in reality any caps, monster beats an awesome assemblage for feathered cloches the fact that regarded a good deal more cutting edge ın comparison to the dresses.

Unknown said...


I enjoyed reading your articles. This is truly a great read for me.
I have bookmarked it and I am looking forward to reading new articles.



wholesale straw hats


Thegamesurplus said...

Thank you for sharing superb informations. Your website is very cool. I’m impressed by the details that you’ve on this site. It reveals how nicely you perceive this subject. Bookmarked this website , will come back for more articles. compare video game price uk

clipping path said...

This is really interesting, You’re a very skilled blogger. I have joined your rss feed and look forward to seeking more of your fantastic post. Also, I have shared your web site
in my social networks! clipping path

Unknown said...

Oh i am really shocking, this post is very nice. I really enjoy it. Thank you for sharing with us. Thanks again very much.

Neck Joint Service/Clipping Path Service/photo retouching Service/Image Masking Service

Unknown said...

Posts shared useful information and meaningful life, I'm glad to be reading this article and hope to soon learn the next article. thank you
Signature:
download free descargar whatsapp plus and download baixar whatsapp plus online and descargar whatsapp , baixar whatsapp gratis

Unknown said...

Many did not hold her tears dare not even see the end of this paragraph
entrar a facebook , facebook en español , entrar no facebook , go girl games , facebook entrar , facebook entrar , facebook entrar

Anonymous said...

youtubelampung.blogspot.com
bimbellampung.blogspot.com
bateraitanam.blogspot.com
lampungservice.com

Anonymous said...

youtubeyoutubeyoutubelampungiklantokoJasaKursuslampungyoutube

photo retouching service provider said...


Awesome creation. Great work. Thanks Photo Retouching Services

Field engineer said...

hi