Unrealized reviews symposium

The One That Got Away

Prior to departing 1UP.com, I prepared two sections for a symposium that never got off the ground. Newsweek's N'Gai Croal, WhatTheyPlay.com's John Davison, and I agreed to generate questions for eight episodes:


Review Scores
Reader Backlash
Reviews in the Age of Social Media
Reviews in the Mainstream Media
Casual, Indie, and User-Generated Games
Review Ethics
Reviews vs. Criticism
Evolving the Review

Sadly, the symposium hasn't happened -- blame me if you must. Below are the questions I suggested for the Review Scores and Reviews vs. Criticism sections. In addition, I've attached a clip from an early email exchange on the topic.


Review Scores
Question 1: How much is on our minds before we begin playing any given game for review purposes? Will we imagine a range of probable scores that a heavily marketed, highly budgeted, and hugely anticipated game will get? What when the game is branded “budget” or is the work of a lesser-known, less-storied studio? If so, how closely have actual scores correlated with our assumptions?

Question 2: Ought reviewers settle on a score before, during, or after writing a review? How consistent are our practices with our prescriptions? Have we, for instance, revised a score after writing our reviews, even though we advocate against it, and if so, why?

Question 3: When possible, do we look at the scores that other critics give to the games that we're reviewing, as we review them? If so, are groupthink or iconoclasty potential problems?

Question 4: Often times we will have repeatedly played and/or previewed games in development prior to reviewing them. Does this familiarity with a particular game's developmental process influence the scores that we assign to the final product in the way that a professor will take into consideration her students' limitations and proven potential when she evaluates papers at the end of the semester?

Question 5: Review writing carries real consequence, especially among members of the enthusiast press. Once-warm PR people and game producers can become cold upon our publication of undesirable review scores, diminishing or eliminating our ability to secure subsequent interviews and access. Postmortem discussions and exclusive looks at the publisher and/or developer's forthcoming products are less likely. Conversely, a few publishers will permit us to post reviews before competitors, provided our review scores are favorable. Do such pressures produce a subliminal background or even enter our thoughts as we write reviews and assign scores?

Question 6: Is grade inflation an ongoing problem?

Question 7: Do scores determine our tone? Can a “3” encourage us to explain an aspect of a game in clearly negative terms where our attitude is actually less decided? Example: Game X's camera obscures the action, combat is irritatingly difficult, and “save” stations are few and far between. In our reviews, is Game X's plot, which we're still thinking through, more likely to become miserable than plain?

Question 8: Do scores encourage our readers to conduct a sort of text-to-number calculus where the two obviously negative statements in an otherwise positive-sounding review necessarily translate into every point deducted from the “10” that the game didn't get? Does this make reviews with high marks more likely to overlook fault, and reviews with low marks less likely to celebrate accomplishment?

Question 9: Which is more important to us, our scores or our copy? If the latter, have our responses revealed any inconsistencies between our attitudes and actions? Are we still convinced of the importance and power of scores?

Related suggestions for Ethics section:
Have we ever submitted review scores to publishers prior to their publication? If so, why?

Have we ever submitted review copy to publishers prior to its publication. If so, why?

Have PR people suggested that specific critics review specific games? Have we complied with their suggestions?


Reviews Vs Criticism
Question 1: What is the object of a review? What are the review writer's obligations?

Question 2: If the purpose of a review is to suggest to consumers how they should spend their time and money, why do we avoid less-granular grading scales such as Buy, Try, or Avoid? Example: Giant Bomb founder and former Gamespot editorial director Jeff Gerstmann told MTV's Multiplayer blog that “'How can I save people money today?' is basically the kind of mentality that I tackle this stuff with.” Under Gerstmann's directorship, Gamespot reviewed games on a hundred-point scale. Is a 9.6 different than a 9.7 when the wisdom of a purchase is what the reviewer wants to communicate?

Question 3: Actual sales rarely correlate with review scores in cases where games are not also heavily hyped and marketed. Increasingly, gamers pre-order games prior to the publication of reviews. Interactive demos allow our audiences to decide for themselves whether or not a game will be worth their dollars. In addition, word of mouth and message board discussions inform our potential audiences' purchasing decisions with an intimacy and directness that we cannot provide. Finally, review aggregation sites such as Metacritic mute the bias of individual reviewers and provide a bigger picture. Do these circumstances suggest that our self-perception is, well, delusional – a throwback to a time when magazines and websites were gaming's gatekeepers? If our audiences believe this, even if we do not, what are they really reading for?

Question 4: Can criticism (concerned with telling our audiences what they're spending time and/or money playing as opposed to whether or not a game is worth spending time and/or money to play) coexist with reviews? Is a competent review also a critique -- as is so often the case where lit, movies, and music are concerned -- or should we separate the two?

Question 5: What can (or should) such criticism take into account? [Note: I don't want to jump the gun on the Evolving Reviews section here, so bear with me if you're wondering why I'm not yet asking certain obvious questions about the shape and challenges of videogame criticism.]

Email snippet:

Perhaps the ethics angle is unavoidable -- at least as it ties into larger concerns about what a review intends to accomplish. Look at the latest bit with Metal Gear, where what all message board posters really reacted to was the irrelevance of the game's install issues. Meanwhile, reviewers agreed to talk around the fact that the game's content is at least fifty percent non-interactive (by most accounts). How can you either review or critique the game under these compromised conditions? This is even bigger than issues outlined in Stephen Totillo's Bill of Rights. Which leads to another concern of mine that might make sense to tackle in the "reviews vs. crit" or "reviews in the age of social media" sections....

Spoiler-phobia leads to a lack of substantiation. Because we won't cite specifics, we're all fluff. N'Gai, you must be familiar with the journalistic notion of a ladder of abstraction -- where concrete, specific details are at the bottom (showing), and the most general or abstract ideas are at the top (telling). The problem is that with reviews we don't reach high enough, nor do we come right down to the bottom. In the mucky middle, we get IGN's reviewer telling us that MGS4 "recast[s] the entire series as nothing less than a re-telling of the Messiah story." As for convincing evidence to argue that the allusion isn't cockamamie? That would be spoiling the story. BioShock and Portal reviews ran into trouble here, too.

I also expect influence and propriety to play a part in the "too harsh/too soft" section. There are real repercussions associated with writing negative reviews. Whether or not we're writing with future exclusives (or even normal, non-exclusive access) immediately on our minds, it's there. In addition, I'm convinced that some inexperienced critics imagine an acceptable range of scores for any given game, even prior to playing. When the press sheet says "budget series", we know it's no holds barred. And finally, we have PR teams whispering in reviews editors' ears that this reviewer is ideal or unsuited to that review assignment.

The sample topics seem suited to allow diversions in any direction I'd want to go with one or two exceptions. Are we interested in talking about craft at all? I'm disturbed by the fact that the enthusiast press hires gamers first and writers/thinkers second. Not once in my career have I sat in on a serious discussion about how we do what we do. Compare this to newspapers with full-time writing coaches.