You Can’t Fake Quality, But That Never Stops Them from Trying
Several years ago, I worked in a development department of a major publisher. For its recent holiday season, it had, with a couple isolated exceptions, mostly released a cavalcade of licensed-IP, rushed and otherwise lackluster titles (sound familiar?). At a subsequent all-hands meeting of the corporate headquarters, the president of the company took the stage and thanked us for our tireless efforts. The company, he said, was doing better than ever before; we had beat earnings expectations, shown positive revenue growth, and pleased Wall Street once again. But not everything was rosy: we had proven time and again that we could make money, but our products were getting pummeled in the press – hurting the company’s brand in the process. This, he said, had to change.
The president outlined his vision: now that we had tackled the basic business of making and releasing games, our next big focus was to be quality. It wasn’t enough to just get the titles on store shelves and into people’s homes. We also had to ensure they were good games. This meant taking a hard look at titles in development and deciding whether or not to continue with them. It meant being careful when choosing partners with which to work. And above all, it meant raising the reputation of the publisher’s name, so that when people saw its logo on the box they knew they wouldn’t be getting something broken.
Being young, naïve, and with no sense of irony or sarcasm at the time (perhaps I exaggerate), I bought into this speech. I was impressed that he had candidly admitted that many of the games his company had shipped were pretty much terrible. He mentioned the importance of making choices and doing things in which one could believe. Finally, he’d pointed out that making high-quality games was good business besides. His words were resonating with me, and I walked out of the meeting hopeful that, if not immediately, in a few years I might be working at the kind of place that wasn’t just an industry player by dint of size but also respect and perhaps one day even cultural cachet.
A while later, I was having lunch with an acquaintance who was a veteran of the company and the industry at large. He had been with this particular publisher through its darkest and tiniest days, and had been in the game industry long enough to have worked on titles that were stored entirely in a couple meager kilobytes. The subject of our conversation turned to my current and his former employer, and I told him about the speech, saying I felt the president really seemed to understand things. He laughed in response and said, “Matthew, they do this ‘quality’ thing every couple years. It must have happened at least a half-dozen times while I was there. Every time, they say they really need to improve quality and start these big initiatives centered around raising the caliber of their titles. You’ll see.”
* * *
Quality is hard to measure and control even in cases where creative judgment and opinion don’t enter the equation, such as in car manufacturing or construction. But the subjective nature of entertainment, with all its opinions, whims, and fashions, complicates things even further. As William Goldman wrote famously of the film business: nobody knows anything. Just when someone thinks they’ve figured out the secret formula to consistently create great entertainment, the ingredients suddenly collide in a way previously unforeseen. Expectations are dashed, creators we believed in are found to be maddeningly inconsistent. Maybe audiences have just gotten bored and are waiting for something fresh and unexpected to take the day. The rules change all the time.
In order to try to raise the quality bar of its titles, the major publisher I worked for instituted, along with many of its peers in the industry, a system of bonuses for production employees based in part on average game review scores, as calculated by sites like GameRankings and Metacritic. This incentive, in theory, pushed employees to achieve the greatness in our games that the president had identified as our goal. But this meant equating a Metacritic score, which averages the numerical scores of dozens of reviews in diverse publications, to real intrinsic quality.
Armed with the knowledge that higher review scores meant more money for them, game producers were thus encouraged to identify the elements that reviewers seemed to most notice and most like – detailed graphics, scripted set piece battles, “robust” online multiplayer, “player choice,” and more, more of everything. Like a food company performing a taste test to find out that people basically like the saltiest, greasiest variation of anything and adjusting its product lineup accordingly, the big publishers struggled to stuff as much of those key elements as possible into every game they funded. Multiplayer modes were suddenly tacked on late in development. More missions and weapons were added to bulk up their offering – to be created by outsource partners. Level-based games suddenly turned into open-world games.
Before you cry in despair, keep in mind that all these people wanted in the end was the best game possible – or, more precisely, the best-reviewed game possible.
* * *
The president who made the big speech about quality left the company a few years later (as did I). His successor might have realized, or known all along, that low-quality movie-license games that happen to be on store shelves on the day of the movie release often make significantly more money than great games based on older properties, because that’s what the company continued doing, to the benefit of its financial performance. Abetted by well-executed sequels to its own properties, it remains to this day a great business success story.
Microsoft, for its part, recently announced that low scores from Metacritic will be part of the criteria used to remove titles from availability on its Xbox Live Arcade download service. I can easily imagine the meeting that took place in Redmond as they rolled out a strategy to focus on quality.
Commentary | May 23, 2008
Reader Comments (16)
Like the ill-conceived production bonuses for Metacritic performance that you mention, the quest for executive-managed, scientifically derived quality also rolls downhill onto content creators in the form of production-driven "quality-enhancing" tactics that fail in a similarly spectacular fashion. I can think of a couple: heavy-handed, production-driven design that turns content creators into mindless implementers; and everyone's favorite, implied/requested crunch ("we're asking you to work [because we can't legally force you to]").
Taking these measures in the name of "quality" doesn't inspire passionate developers to work harder - it demotivates them, working against the intented goal of releasing a high-quality game. The same old problems of big-studio development just persist under a banner that seems really fake to people who actually do care about the game's quality from an enthusiast's/craftsman's standpoint.
I think the problem usually goes like this: People in charge honestly do want to make a quality game. Motivations might differ from what other people on the team wants, but at the end of the day, everyone wants a good game. However, managers don't understand that because they say they want quality necessarily doesn't make it so. After not getting immediate "quality" (a totally subjective term by the way), managers start making irrational decisions, at which point the project enters a tail-spin.
Manager types generally get what they want by saying it: "Free up some memory," "Put more engineers on the physics system," "Add more destructible objects," are all things I've heard, while everyone else on the team says "Yes Sir!" and makes it happen.
That's not the case with "Let make a quality game!" If that were true every game would be mind blowingly awesome. Project managers sometimes don't understand that there's more involved in getting quality games from a team: making sure the team in heavily invested in the project, the team is communicating and working together, the team is doing rapid iterations on every facet of the game, and so on. Even under those conditions it's possible to make a game that isn't stellar.
Anyway, once again, great post and a great comment from Sean as well.
So, the push for quality isn't an internal directive, but an external one, affecting how long thrid-party titles get to stay in the marketplace. Funny, though, considering XBLA was supposed to be indie-developer friendly. I guess that's only the case if you're indie AND popular.
Great post, though. I personally have a lot of issues with using Metacritic as anything other than a review aggregator, and yet everyone keeps treating it like the Holy Effing Bible of Game Quality.
Viewing a list of games that could potentially get pulled, ( http://en.wikipedia.org/wiki/Xbox_Live_Arcade click 'show') you will see a lot of games from the big publishers- Namco, Atari, PopCap, etc.
In fact, this strongly targets (rightly so) the large amount of retro games put out the big publishers trying to squeeze a few more dollars from their existance.
Metacritic to turn 'subcription only' and start charging money for memberships.
"If you prick game publishers are going to base developers income on our average scores, then you can pay us some of that coin to see them!"
That's what I'd do anyway ;-)
All through the ages, Quality in and of itself has been a very hard idea to define and measure. In games, I feel that quality has so many different meanings that reviews should be taken with a grain of salt and not be used as the sole basis for bonuses etc.. What floats one gamer's boat might not float another. In the same token, there is no single person (aka reviewer) that has the same tastes as everyone.
Proper segmentation of the target audience and defining what their idea of a quality is a good first step in improving quality.
There are specific reasons or qualities why gamers choose to play certain games and reasons why should be taken into consideration. Why play a casual game when they have 10 tripple A top notch titles to choose from? People play casual games becuse of the low time commitment, another important factor would be that it's easy to pick up and addictive. Finding the fine line between challenging and playability is difficult to assess especially if you are designing an improvement to a game which is supposed to appeal to a large percentage of your target audience which leads into who is the target audience? What characteristics make up the matrix of the 60% of the 30/60/10 rule? etc.
Quality is subjective and there are many metrics that make up what it means to be a quality game.
Unfortunately the people calling the shots didn't know jack about quality (in terms of either software engineering or entertainment), as they would prove beyond measure. The millions of dollars I saw wasted was mind boggling. The heartbreaking part was how talented the people on the team were (well, maybe a hundred of them were), and seeing them disheartened, slandered, and abused by taking advantage of their creative drive to make a 'quality' game. Attrition was huge on the project and beyond, though the leadership didn't (and still doesn't) think it is a big problem.
The last slap in the face is this unswerving allegiance to metacritic. It is such a stupid tact to put so much power in the hands of mostly incompetent, half illiterate, and almost completely unprofessional schmoes who are game reviewers. These people, on average, love shooters and boobs (GTA a 99? riiiight) and not much else (except maybe twinkees and skipping showers). The bias, vitriol, rabble rousing, and corruption of some of them is despicable and some industry leaders want to base their business on it? In the end metacritic is a poor man's substitute for actually playing games, and without playing the latest games execs will keep making ill-informed decisions about the competition (aka the quality bar) are basically scuttling their businesses on the backs and reputations of the naive (aka me a few years ago).
But for publishers to use it as the basis for bonuses, etc., is crazy, because ultimately it's just an averaging of opinions! I dig things that MC claims are lame, just as MC renders great verdicts on things that I think are terrible. I think that the key for publishers should be to get their marketing departments to work on initiatives that get the product into the hands of consumers on a trial basis - because most folks are unswayed by critics when they already have formed an opinion on something.