Magical Wasteland is a home for writings by a certain Matthew S. Burns.

Other things: 

Recent Comments
Shadegrown Games

Updates from Shadegrown Games:


Due to popular demand, you can now order a tasteful Magical Wasteland t-shirt

« A Secret Treasure of Okami | Main | The Best Opera Yet Written »

You Can’t Fake Quality, But That Never Stops Them from Trying

Several years ago, I worked in a development department of a major publisher. For its recent holiday season, it had, with a couple isolated exceptions, mostly released a cavalcade of licensed-IP, rushed and otherwise lackluster titles (sound familiar?). At a subsequent all-hands meeting of the corporate headquarters, the president of the company took the stage and thanked us for our tireless efforts. The company, he said, was doing better than ever before; we had beat earnings expectations, shown positive revenue growth, and pleased Wall Street once again. But not everything was rosy: we had proven time and again that we could make money, but our products were getting pummeled in the press – hurting the company’s brand in the process. This, he said, had to change.

The president outlined his vision: now that we had tackled the basic business of making and releasing games, our next big focus was to be quality. It wasn’t enough to just get the titles on store shelves and into people’s homes. We also had to ensure they were good games. This meant taking a hard look at titles in development and deciding whether or not to continue with them. It meant being careful when choosing partners with which to work. And above all, it meant raising the reputation of the publisher’s name, so that when people saw its logo on the box they knew they wouldn’t be getting something broken.

Being young, naïve, and with no sense of irony or sarcasm at the time (perhaps I exaggerate), I bought into this speech. I was impressed that he had candidly admitted that many of the games his company had shipped were pretty much terrible. He mentioned the importance of making choices and doing things in which one could believe. Finally, he’d pointed out that making high-quality games was good business besides. His words were resonating with me, and I walked out of the meeting hopeful that, if not immediately, in a few years I might be working at the kind of place that wasn’t just an industry player by dint of size but also respect and perhaps one day even cultural cachet.

A while later, I was having lunch with an acquaintance who was a veteran of the company and the industry at large. He had been with this particular publisher through its darkest and tiniest days, and had been in the game industry long enough to have worked on titles that were stored entirely in a couple meager kilobytes. The subject of our conversation turned to my current and his former employer, and I told him about the speech, saying I felt the president really seemed to understand things. He laughed in response and said, “Matthew, they do this ‘quality’ thing every couple years. It must have happened at least a half-dozen times while I was there. Every time, they say they really need to improve quality and start these big initiatives centered around raising the caliber of their titles. You’ll see.”

* * *

Quality is hard to measure and control even in cases where creative judgment and opinion don’t enter the equation, such as in car manufacturing or construction. But the subjective nature of entertainment, with all its opinions, whims, and fashions, complicates things even further. As William Goldman wrote famously of the film business: nobody knows anything. Just when someone thinks they’ve figured out the secret formula to consistently create great entertainment, the ingredients suddenly collide in a way previously unforeseen. Expectations are dashed, creators we believed in are found to be maddeningly inconsistent. Maybe audiences have just gotten bored and are waiting for something fresh and unexpected to take the day. The rules change all the time.

In order to try to raise the quality bar of its titles, the major publisher I worked for instituted, along with many of its peers in the industry, a system of bonuses for production employees based in part on average game review scores, as calculated by sites like GameRankings and Metacritic. This incentive, in theory, pushed employees to achieve the greatness in our games that the president had identified as our goal. But this meant equating a Metacritic score, which averages the numerical scores of dozens of reviews in diverse publications, to real intrinsic quality.

Armed with the knowledge that higher review scores meant more money for them, game producers were thus encouraged to identify the elements that reviewers seemed to most notice and most like – detailed graphics, scripted set piece battles, “robust” online multiplayer, “player choice,” and more, more of everything. Like a food company performing a taste test to find out that people basically like the saltiest, greasiest variation of anything and adjusting its product lineup accordingly, the big publishers struggled to stuff as much of those key elements as possible into every game they funded. Multiplayer modes were suddenly tacked on late in development. More missions and weapons were added to bulk up their offering – to be created by outsource partners. Level-based games suddenly turned into open-world games.

Before you cry in despair, keep in mind that all these people wanted in the end was the best game possible – or, more precisely, the best-reviewed game possible.

* * * 

The president who made the big speech about quality left the company a few years later (as did I). His successor might have realized, or known all along, that low-quality movie-license games that happen to be on store shelves on the day of the movie release often make significantly more money than great games based on older properties, because that’s what the company continued doing, to the benefit of its financial performance. Abetted by well-executed sequels to its own properties, it remains to this day a great business success story.

Microsoft, for its part, recently announced that low scores from Metacritic will be part of the criteria used to remove titles from availability on its Xbox Live Arcade download service. I can easily imagine the meeting that took place in Redmond as they rolled out a strategy to focus on quality.

Reader Comments (16)

It's certainly a great idea to strive for quality - personal, collective, ultimate. The relentless pursuit of perfection, Lexus-style. What makes this ring hollow when spoken from the mouths of high-level executives (generally not really consumers of the products they're trying to improve), I think, is in their attempts to execute that goal. However, I think the funny thing about it is that the means taken in the quest to achieve this end of releasing a quality game has a lot more to do with how the game turns out than the goal itself.

Like the ill-conceived production bonuses for Metacritic performance that you mention, the quest for executive-managed, scientifically derived quality also rolls downhill onto content creators in the form of production-driven "quality-enhancing" tactics that fail in a similarly spectacular fashion. I can think of a couple: heavy-handed, production-driven design that turns content creators into mindless implementers; and everyone's favorite, implied/requested crunch ("we're asking you to work [because we can't legally force you to]").

Taking these measures in the name of "quality" doesn't inspire passionate developers to work harder - it demotivates them, working against the intented goal of releasing a high-quality game. The same old problems of big-studio development just persist under a banner that seems really fake to people who actually do care about the game's quality from an enthusiast's/craftsman's standpoint.
May 23, 2008 | Unregistered Commentersean
Wow. Great post. You really hit the nail on the head, especially since I've been in multiple situations that you describe and can confirm its accuracy.

I think the problem usually goes like this: People in charge honestly do want to make a quality game. Motivations might differ from what other people on the team wants, but at the end of the day, everyone wants a good game. However, managers don't understand that because they say they want quality necessarily doesn't make it so. After not getting immediate "quality" (a totally subjective term by the way), managers start making irrational decisions, at which point the project enters a tail-spin.

Manager types generally get what they want by saying it: "Free up some memory," "Put more engineers on the physics system," "Add more destructible objects," are all things I've heard, while everyone else on the team says "Yes Sir!" and makes it happen.

That's not the case with "Let make a quality game!" If that were true every game would be mind blowingly awesome. Project managers sometimes don't understand that there's more involved in getting quality games from a team: making sure the team in heavily invested in the project, the team is communicating and working together, the team is doing rapid iterations on every facet of the game, and so on. Even under those conditions it's possible to make a game that isn't stellar.

Anyway, once again, great post and a great comment from Sean as well.
May 25, 2008 | Unregistered CommenterJesse Snyder
The problem with your analogy, though, is that unlike Acti-Blizzard-Ubi-EA, titles on the XBLA are not developed BY Microsoft. They are typically third-party titles. It's only VERY recently that MS even started an XBLA studio.

So, the push for quality isn't an internal directive, but an external one, affecting how long thrid-party titles get to stay in the marketplace. Funny, though, considering XBLA was supposed to be indie-developer friendly. I guess that's only the case if you're indie AND popular.

Great post, though. I personally have a lot of issues with using Metacritic as anything other than a review aggregator, and yet everyone keeps treating it like the Holy Effing Bible of Game Quality.
May 27, 2008 | Unregistered CommenterGeorge
and relying on metacritic so much on the publisher side starts to exert pressure on the PR plan for the company's games in a (often times misguided) attempt to sway reviewers. i know that everybody already has opinions about this, and some are right, some are wrong, and some only apply to certain outlets or journalists, but the net effect is Not Good no matter what.

May 27, 2008 | Unregistered Commenterferricide
Good post. Very good read. I especially liked the comment about "more is better". Sometimes more isn't better. In Ghost Recon: Advanced Warfighter, for example, there are so many guns that are almost identical I get more confused than impressed.
May 29, 2008 | Unregistered CommenterOsama
Microsoft may be basing whether or not an XBLA game gets pulled by the Metacritic review. True. I don't think this makes it less of a haven for Indie developers.

Viewing a list of games that could potentially get pulled, ( click 'show') you will see a lot of games from the big publishers- Namco, Atari, PopCap, etc.

In fact, this strongly targets (rightly so) the large amount of retro games put out the big publishers trying to squeeze a few more dollars from their existance.
May 29, 2008 | Unregistered CommenterNoodle of Death
The next logical step here?

Metacritic to turn 'subcription only' and start charging money for memberships.

"If you prick game publishers are going to base developers income on our average scores, then you can pay us some of that coin to see them!"

That's what I'd do anyway ;-)
May 29, 2008 | Unregistered CommenterShane
Shane: IMO you're thisclose to being exactly right. I don't think Metacritic will start being subscription-based to consume their aggregate reviews, instead they'll start a pay-for-membership model where publishers/developers have to pay them to have Metacritic display their work at all. When Metacritic becomes as indispensable to the industry as to base pay scales on them, they'll have the power to do this and the publishers will pony up whatever they're asked to.
May 29, 2008 | Unregistered CommenterJT
I think that since there is a lack of reliable measure of QUALITY in the industry in general, Metacritic has risen to the top thus is being used as the holy bible of games (and other stuff). You can't take sales as the sole indicator of quality just like how you can't take reviews as the sole indicator of quality (even aggregated and averaged reviews)

All through the ages, Quality in and of itself has been a very hard idea to define and measure. In games, I feel that quality has so many different meanings that reviews should be taken with a grain of salt and not be used as the sole basis for bonuses etc.. What floats one gamer's boat might not float another. In the same token, there is no single person (aka reviewer) that has the same tastes as everyone.

Proper segmentation of the target audience and defining what their idea of a quality is a good first step in improving quality.

There are specific reasons or qualities why gamers choose to play certain games and reasons why should be taken into consideration. Why play a casual game when they have 10 tripple A top notch titles to choose from? People play casual games becuse of the low time commitment, another important factor would be that it's easy to pick up and addictive. Finding the fine line between challenging and playability is difficult to assess especially if you are designing an improvement to a game which is supposed to appeal to a large percentage of your target audience which leads into who is the target audience? What characteristics make up the matrix of the 60% of the 30/60/10 rule? etc.

Quality is subjective and there are many metrics that make up what it means to be a quality game.
May 30, 2008 | Unregistered CommenterGüd Bier
Güd your exactly right about Quality. Everyone claims to have the answer but in reality it's very subjective and varies person to person and that divide grows as the amount of people increases. Be they the developer, the publisher, and everyone in between.What I find more disturbing than basing quality on a system of review scores and sales numbers, is the sheer amount of outspoken people both in the industry and outside it who claim to have the answer to quality. Or expect faith/respect on their single-minded ability to fulfill consumers needs/wants purely because of "who they are", and as a result often lash out at people who disagree with their "vision".My mantra is everyone wants to make a great game, and everyone thinks they have the answer to making a great game. The reality is no one has the answer or if they do it's fleeting.As such you should weigh any quality decisions carefully and realize the significance of it's impact. Look at your decisions from all angles, try and get a consensus from as many people as you can and as many sources you can. Make an educated decision not an opinionated decision.The key is balance.

May 30, 2008 | Unregistered CommenterMichael
Very well said. I was 'lucky' enough to get into the industry right when one of the more notorious 'bad games' was entering production, and was part of a large team that got ran through the mud (almost a year of delay and crunch) in the name of quality.

Unfortunately the people calling the shots didn't know jack about quality (in terms of either software engineering or entertainment), as they would prove beyond measure. The millions of dollars I saw wasted was mind boggling. The heartbreaking part was how talented the people on the team were (well, maybe a hundred of them were), and seeing them disheartened, slandered, and abused by taking advantage of their creative drive to make a 'quality' game. Attrition was huge on the project and beyond, though the leadership didn't (and still doesn't) think it is a big problem.

The last slap in the face is this unswerving allegiance to metacritic. It is such a stupid tact to put so much power in the hands of mostly incompetent, half illiterate, and almost completely unprofessional schmoes who are game reviewers. These people, on average, love shooters and boobs (GTA a 99? riiiight) and not much else (except maybe twinkees and skipping showers). The bias, vitriol, rabble rousing, and corruption of some of them is despicable and some industry leaders want to base their business on it? In the end metacritic is a poor man's substitute for actually playing games, and without playing the latest games execs will keep making ill-informed decisions about the competition (aka the quality bar) are basically scuttling their businesses on the backs and reputations of the naive (aka me a few years ago).
May 31, 2008 | Unregistered CommenterDesigner4Quality
To make good games, the designers and developers need to be freed from arbitrary movie-release schedules and they need to believe passionately in what they are making. Thats very unlikely to happen w/ a movie-based game, where they don't get to choose the characters or typically much of the plot. Blizzard and Id are examples of two gaming houses that release products "when they are ready" and not according to arbitrary market deadlines, and the quality shows brightly, and they have made more money than these cheeseball movie-game cookie-cutter makers.
May 31, 2008 | Unregistered CommenterZeevious
i wonder what company this is about. hmm..
June 10, 2008 | Unregistered Commenterweb design company
You know, the shame is that I really heart Metacritic. I loved the idea when I first saw it in college, and I still use the site almost every day to get a sense of - ruh roh - whether an entertainment product is worth my dollars. Just the concept is cool and was super unique when it came out - get a sense, in general, of what people think of a subjectively judged product. In this respect I think it still fulfills its mission.

But for publishers to use it as the basis for bonuses, etc., is crazy, because ultimately it's just an averaging of opinions! I dig things that MC claims are lame, just as MC renders great verdicts on things that I think are terrible. I think that the key for publishers should be to get their marketing departments to work on initiatives that get the product into the hands of consumers on a trial basis - because most folks are unswayed by critics when they already have formed an opinion on something.
June 15, 2008 | Unregistered Commentersean
The only rule I know about quality is this: it's made by talented people. So to try to find some external criteria about it is foolish. But what isn't foolish is to find and invest in the people with talent. Not just companies. People.
August 21, 2008 | Unregistered CommenterGrassroots Gamemaster
The problem isn't just that Metacritic is only an average of opinions - it's an average of the opinions of a limited selection of game critics, a minuscule subsection of people, whose minds these same companies actively attempt to skew with shady backroom deals. People who are enjoying less consumer trust with every Gerstmanngate that happens.

August 23, 2008 | Unregistered Commentersamperi

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>