Guest Post: Andy Patrick
Recently I was asked the question, “is crunch necessary to produce a AAA game?” My response is very simple: “No, and I know this for a fact.” I know this, because I have done it, and if you’re interested I’d like to share with you what I did.
It’s a story in two parts – the first part is about a game project on which I crunched, to explain what it is for readers who don’t know; the second part is about a game project on which I did not crunch. This second part is much more important because it’s what allows me to answer the question with such certainty. If you know what crunch is already, you can skip right to it.
Before we get into the story though, some disclaimers, in a rapid-fire-machinegun-bullet-point-style:
- Both parts to this tale are games I worked on at Rare.
- This article is not to be read as a judgement on Rare, but on crunch.
- If I were to write a judgement on my time at Rare, which I won’t, it would be generally favourable.
- I joined Rare in 2002.
- I had five really good years there and left in 2007.
- During that time I worked with many very talented people.
- I know next to nothing about what Rare is like now.
- Even if I did know anything about what Rare is like now, this article is not intended as a judgement on Rare.
- I wrote this article from memory. Details here and there may not be perfectly accurate, but I’ve done my best.
Crunch – What Happened?
Shortly after joining Rare I was moved onto the Kameo team. This game was originally slated for GameCube and had already been in development for several years; but not very long after I joined the team, Microsoft bought Rare. The decision was made to move Kameo to Xbox. Rightly or wrongly, Xbox would become known as a game for the “hard-core gamer”; it soon became obvious that on Xbox, Kameo would sink without trace. Thus the decision was made to make Kameo an Xbox 360 launch title.
In early summer 2005, it became evident that at the rate we were going, Kameo wouldn’t be ready for launch. We were told this in a team meeting, but I think we all knew it. If we missed launch, we’d miss Christmas. If we missed Christmas, we’d disappear among all the Xbox 360 games due out the next year. It was as simple as this: we were going to have to work harder. So we did.
For the next four months or so, I was in full-on crunch mode. People have crunched for longer than four months; and although I regularly worked 80 hours a week, approaching 100 hours in a week sometimes, people have worked more hours than that too. I was by no means the only one on the team to do this – far from it, I think we all crunched to a greater or lesser degree. Indeed, I never spent more than 24 hours straight in the office; I know for certain that one or two people on the Perfect Dark Zero team (also to be a launch title) did exactly that. I wasn’t special, but I was crunching – hard.
We missed a lot of deadlines in the death-march towards RTM (Release To Manufacturing) but we pulled it together, worked ever-longer hours and hit the most important deadline of all, RTM itself. Kameo released as a launch title on the Xbox 360, and so did Perfect Dark Zero. I may not be the most appropriate person to judge a game I poured so much of my life into, but I think Kameo is an awful lot of fun and it received a generally favourable response. It’s not without its flaws, but the combat is very enjoyable and some of the environments, music and sound effects are in my opinion staggeringly beautiful. I am proud of my part in creating it.
Crunch – How Did It Affect Me?
Despite my pride in the finished product, there’s a dark cloud to go along with the silver lining.
For those four months of summer, I barely saw the sun. I’d get to work before nine, and work straight through until five, eating lunch at my desk. Having worked a solid eight-hour day already, I’d give myself a break for half an hour then work another seven- or eight-hour day. It was usually past midnight by the time I left to drive home – though to be fair, if it was very much past midnight, I’d be told I didn’t need to be in for nine the next morning.
Weekends didn’t exist; Saturdays and Sundays were spent like any other day: sat at my desk, writing code and fixing bugs. And I wasn’t alone. We all crunched to a greater or lesser degree. On the days we had to produce a build to give to QA, nobody could go home until the build was complete, in case a problem was found and they needed to be there to fix it. The build process took six hours and was rarely started before 6pm.
I have never been slim, but during this time I got fat, or possibly just fatter. I didn’t have time to exercise, and was never at home to cook for myself. Rare has an extremely good restaurant, but I usually ate crisps and chocolate at my desk. In the evening, the producer bought the team pizza. I haven’t got the best relationship with food and usually ate more than my share (anything other people didn’t want). By the time the game was finished I’d gained two stone and none of my clothes fit properly any more.
Crunch made me stressed. I’m usually quite a laid back guy, but at one point, one of the other coders asked me to help them out with something while I was eating a bag of crisps during my evening break. I shouted at him, and it’s something I’m not proud of. I told him how many hours I’d already worked that week and it was only Wednesday. (I wasn’t alone in that of course). I think I may have sworn at him, and I was definitely very rude, unnecessarily so. Not the proudest moment of my life, by a long shot.
I got ill, too. I can’t prove it was a direct result of crunch (the doctor said it was “just a [non-specific] virus”, though I hadn’t been anywhere but work that might have given me a virus) but for three days I could do nothing but shiver, sweat and soak my head in a cold cloth to keep the headaches away. On the fourth day I started to believe I wasn’t going to die; but I knew they’d found a bug in my code, so still shivering a bit and sweating a bit (but with hardly any headache) I went back to work.
Finally, crunch nearly killed me. Think I’m exaggerating? I had a weekend off – if I remember right, the only weekend off I had all summer – to attend a wedding near to where my parents live. On the Friday evening, I drove the 200 miles to my parents’ house, got in the front door and fell asleep on the floor in the hallway. It is a miracle I hadn’t fallen asleep at the wheel, crashed and died (or, worse, killed someone else). Another mile’s worth of driving and I’m certain I would have.
All those of you who think crunch is OK – especially if you’re a manager, inflicting it on other people – tell me you still think it’s OK having read all that. Go on, I dare you.
So, what did I get for all this? Well, more than most on the team actually. I was lucky enough to be one of the ones that Microsoft sent over to LA for the Xbox 360 launch party, which was a pretty awesome experience. Apart from that, I got what everyone else got. Kameo had been a moderate success (last I heard it sold about 950K copies, though sales had slowed to zero) so we did get moderate royalties payments. I can’t tell you for sure how many hours I worked, so I can’t tell you my effective hourly rate if you assume the royalties “paid” for my overtime; I think it would have been more than minimum wage, but it would definitely have been less than simply paying me for my time.
So sixteen weeks of crunch had made me fat, stressed, ill, tired, and nearly dead, plus I had no social life to speak of. In return I was slightly more wealthy than most people my age and was the proud owner of an exclusive Xbox 360 launch team ‘05 faceplate. I took a long hard look in the mirror, and resolved never to crunch again.
No Crunch – What Happened?
The next game I worked on was Viva Piñata. (Technically, I worked on Banjo first, then Viva Piñata to completion, then Banjo again; but Banjo wasn’t released by the time I left Rare). It was released approximately one year after Kameo, and this time I recorded exactly how many hours of overtime I worked in total in that year. How many hours, you ask? Twelve.
There are a number of reasons for this success, and by no means were all of them the result of my own endeavours. Firstly, Kameo had had a long and painful gestation period; moving from GameCube to Xbox, and Xbox to 360, had basically completely invalidated the design, the schedule, most of the art assets and some of the code, each time. Piñata had gone through some similar changes, but each time it happened for Kameo it had supposedly been “six months away from release”, so it was never deemed worth doing things properly and starting again – some of the code in Kameo was six years old, and (though it may seem strange) code rots. Kameo was rotten to the core. For Piñata, the transitions had been smaller scale, with a smaller team, and they’d had (or, perhaps, had taken) the time to do things right.
For example, when Kameo had begun (before my time, but as I understand it) there had been four programmers on the team. CVS was considered an adequate source control solution for four programmers, and I guess it may have been. By the time Kameo was in crunch, there were over twenty programmers on the Kameo team, with as many again on the engine team, and let’s not begin to talk about how many artists there were. CVS was simply inadequate with the result that it broke. Daily, if not hourly, and each breakage required hours of fixing. The Piñata team, by contrast, were using Microsoft’s source control tool based on Perforce – and a suite of automated tests that mean you only checked things in if you were certain they worked, and if by some chance something still went wrong, you’d automatically be notified of the problem inside five minutes.
On Kameo, producing a build took six hours as I’ve already mentioned, and only myself and one other programmer really knew how to do it (we tried to document it, but there were so many manual steps and so much that could go wrong…). On Viva Pinata, every time you checked in a change, the build suite would automatically pick up that change and start building a new version. Thus, on Kameo, when we needed to produce a build for QA, nobody would be allowed to touch anything for six hours in case they broke it; but on Pinata, when we needed to produce a build for QA, we had one already. Automatically!
In addition the tools in use on Viva Piñata were light years ahead of what we had been using in Kameo. Everything was data-driven, meaning that designers and artists could make changes without programmer involvement, meaning that designers could design and artists could make art and programmers could write code. Sounds pretty simple, but the ability to be able to just get on with your job instead of having to wait for someone else to have time to do what you needed to do for you, made everything ten, twenty times more efficient. Again – this isn’t condemnation of the Kameo team, not by any stretch of the imagination. We didn’t have, or hadn’t taken, the time to make the tools what they needed to be; that’s just how it was. But it crippled us and was a major contribution to why we had to crunch on Kameo.
I effectively didn’t crunch on Viva Pinata; twelve hours of overtime in a year is not crunch.
No Crunch – How Did It Affect Me?
I had made up my mind not to crunch ever again, and I made sure that’s how it worked out.
Understand clearly that I did not achieve this by slacking. On Viva Pinata, I was responsible for the entirety of the audio system and tools (which went on to become the basis of the audio engine intended for use by the whole company), as well as the asset database and parts of the build process. These are big, complex systems; it’s not unusual for companies to have four or five audio programmers alone. Nor did I do a botch job. One of the milestones towards the end of the project is what Microsoft call Zero Bug Release (in theory, the first time the game has no bugs logged against it – which will change as soon as QA next get hold of it, but each milestone from then on should see them finding fewer and fewer bugs); I was the only programmer on the team of more than twenty to actually have zero bugs at the ZBR milestone. I did a very good job as a significant part of the Viva Pinata development team, and I did not crunch at all.
So how did I achieve this? Well, for a start I’d probably taken on too much on Kameo (audio system, in-game camera, save-game system, text and localisation, lip-sync and part of the build process!) but more to the point I started Kameo with zero experience; I now had several years under my belt, and had spent the downtime after the release of Kameo educating myself in programming techniques, especially making sure I learned C++ to a level that most programmers never do. (It’s a complex language, frighteningly so at times). I had become a good programmer, with valuable experience. The software I wrote for Viva Pinata was therefore much better designed and much better written, with fewer bugs and shorter development time.
In fact, the code I wrote for Viva Pinata was among the best I have ever written. What makes good code good, you ask? Well, it was correct, reliable, efficient, feature-complete, readable, maintainable, discoverable, reasonably well-documented (…considering), testable, scalable, secure, and simple in structure but able to achieve complex results. I’m not certain whether the fact I was writing good code meant I didn’t have to crunch, or whether the fact I wasn’t crunching meant I was fresh enough to write good code. In fact, of course, it’s a bit of both. You can’t write good code at the end of a sixteen hour working day. You just can’t. During the development of Viva Pinata, I was able to leave work at 5pm, and arrive at 9am the next morning knowing I could continue working productively, instead of wasting time fixing all the mistakes I’d made the night before.
I was relaxed, I lost weight, and I wrote some of the best code I’ve ever written in my career. Viva Pinata launched on time, reviewed pretty well and sold pretty well too. I got similar royalty payments to what I’d had on Kameo, but without nearly killing myself. Plus I had a social life. Some may say that Viva Pinata isn’t grey-and-brown (or “hard-core”) enough to be a AAA game; I have nothing to say in response because such a statement is self-evidently worthless. It released at the same time as Gears of War and there were a surprising number of reviews saying things like “I got review copies of Pinata and Gears at the same time, and thought I’d better play Pinata first otherwise I’d be too busy playing Gears – but I’ve not been able to stop playing Pinata since!”
In the long run, Viva Pinata may not have caught the attention of the hard-core gamers and specialist press like Gears of War did, but it was a high-budget, high-production value game with a strong IP, large marketing budget (including accompanying TV series) and went on to spawn numerous spin-offs and sequels. I’m not sure if it achieved everything Microsoft wanted of it (attracting kids and the Nintendo crowd to the Xbox 360, basically) – you’d have to ask them. But it was unquestionably a AAA game (whatever that really means!) and I did my part of it without crunch. So it is provably possible to make AAA games without crunch.
You may notice the slight get-out clause there: I didn’t crunch, but yes, some of the team did. That doesn’t invalidate the argument in the slightest. Just because they did, doesn’t prove they needed to; it only proves that they did, nothing else. And understand what I mean by “needed to”. We “needed” to crunch on Kameo, because we’d failed to prepare and left ourselves no other choice. Likewise, there may have appeared to be reasons that some of the other Viva Pinata members “needed” to crunch – and that’s not a criticism of them because, after all, I’d been there myself, the previous year. I assert that it appeared to be like that because they didn’t know there was any other option, or perhaps knew but chose not to take it for reasons of their own. By making use of the excellent tools the team were using, and the sensible scheduling; by preparing correctly and working to a plan, writing good code and not taking on too much; they could have done it without crunch. I know this, because I’ve seen both sides.
I worked on a AAA game, and I produced some of the best code of my career, I did it all to spec and on schedule, and I did it without crunch. Nobody can convince me that crunch is necessary because I have proved that it is not. I think it’s sad that so many try so hard to convince their underlings, their team-mates, their peers and colleagues, and themselves.
(this is a guest post by Andy Patrick – if you have your own experience of Crunch, please get in touch – you can post it directly here (anonymously, if necessary), or write a blog post and we’ll link to it)
