Red Ring of Death

When the Xbox 360 was launched in December 2005, Microsoft had accomplished a significant objective that would give it the edge in the console war - beating Sony's PS3 to market by almost a year. But a few months later, a huge problem began to surface - a major hardware defect that threatened to destroy the entire Xbox brand.

TIM DAVID: So, I was an original Xbox owner, shortly after they released. Like many people, I kind of wasn't too sold on the idea of Microsoft entering the console gaming market. But I slowly came around after it had a few exclusives I was super into. And I was very curious about Xbox Live specifically, because you know, at that point, the idea of playing online on the console was still kind of new.

JAMES PARKINSON: This is Tim David.

TIM DAVID: My name is Tim David. I am an American musician, and Xbox enthusiast.

TIM DAVID: So, I used it to the point where the 360 just kind of felt like a natural progression for me. I'd actually recently gotten my first job, shortly before the 360 was set to launch, at a EB Games in America. And when they started taking pre-orders for them, I knew I was going to have the money, I was, you know, just graduated from high school and living at home, so I didn't have a lot in the way of expenses. And it just seemed like, it was kind of nice to kind of be on the ground floor of a console again. I hadn't owned a console at launch since the Dreamcast that my parents surprised me with when I was, like, 13. And I wanted to just kind of stick with Microsoft, because I liked so much of what they were doing, and I was way more interested in anything that was offering, than whatever we'd seen from the PlayStation 3 to that point.

JAMES PARKINSON: The Xbox 360 was released in North America in December 2005, beating the PlayStation 3 to launch by almost a year. Tim had pre-ordered his console but due to smaller allocations at launch, he had to wait a month or so more. But once he got his hands on the 360, it was game on.

TIM DAVID: And from there, it just kind of became my main gaming device. Like, anything that came out for the 360, I would totally buy for the 360, even if it was on something else that I already had. And I would take it to my friend's houses, because I was the only person that had one for a while and all those, kind of, early adopter perks you tend to get by being the only kid in the neighborhood with the shiniest game console.

TIM DAVID: And you know early on, say what you will about launch lineups, but when it came out, there wasn't anything too incredible for it. But I knew in the back of my mind that like one day there was going to be another Halo, one day there was going to be another Splinter Cell. And I guess it was just kind of me getting the drop on all of those games, right? Like, if I have my Xbox now, I know I'll have an Xbox when all the really big, important stuff starts coming out. And then I'll just play you know, Project Gotham or Perfect Dark Zero or whatever they can give me in the meantime. I think it was just the library more than anything else that kind of attracted me to it, and still does, to a degree.

JAMES PARKINSON: One of those big titles that Tim was looking forward to in 2007 was BioShock, but he unfortunately didn’t get the chance to play it at launch. And of course, you’ve seen the title of this episode...you probably know what’s coming.

TIM DAVID: I remember it was just kind of an average summer day, I had the afternoon off at home, I was playing something else that wasn't Bioshock. And then suddenly, I heard, like, a kind of a really sad, almost coughing noise for my Xbox. Suddenly, it was off and nothing - no signal was coming to the TV. And I tried everything I could but, in the back of my mind I just knew like, “okay, it finally came for me, I was too arrogant and my numbers up”. So I went over to it, I tried turning it back on a couple times. And sure enough, I had gotten the red ring.

JAMES PARKINSON: The original version of the Xbox 360 suffered from a major hardware failure. If the ring light around the power indicator on the front of the console turned red, instead of its usual green, you had a problem. A problem which the internet affectionately dubbed the “Red Ring of Death”. In most cases, the hardware failure would cause consoles to stop working altogether, and the issue was widespread. Like so many others, Tim sent his console back to Microsoft and received a replacement a few weeks later. For some customers, it was problem solved. For others, including Tim, the problem didn’t end there.

TIM DAVID: It didn’t. So that Xbox died maybe a year or two after I got it, which again, unfortunately for ‘red ring’ consoles, still a decent lifespan. And then from there, I would actually go on to go through, for a number of reasons, I would actually go through nine Xbox 360s total.

TIM DAVID: For the most part, they would get exchanged through my job instead of going through the Microsoft process anymore, because I had a warranty on and they would let me exchange them at the store. But I would go through nine Xbox’s total, like counting that first one that I had. They would die for a number of reasons, or I would at least have to return them for a number of reasons. But for the most part, I would say out of those nine, I would say seven of them would ‘red ring’, just as kind of a matter of inevitability through their lifespan. It got to the point where I almost wouldn't even blink when it would happen. Like, anytime I turned my Xbox on I knew today might be the day, and I was just prepared to drive back to the mall and talk to whatever manager was on duty and have him hand me over another one.

James Parkinson: So, through that whole process of, like, you know, another console, another console, what was happening when you were going back to work and bringing your faulty console in each time? Talk me through that, that process and kind of what the staff thought and all that.

TIM DAVID: We were all pretty close, we’re all pretty good friends. And as a result, we all kinda learned to take a joke. So when I would bring them in, it was sort of the same way people would respond to - if a friend of theirs had broken up with somebody, to immediately start dating somebody else, right? Like, they couldn't believe that I was just going through them this quickly. They’d believe me, they would test these things at the store. But it was just a lot of, kind of, loving sneers and disbelief, because they couldn't understand how this kept happening to me. Especially because, I should point out, so many of these happened over the span of a single summer. Like, there was a three or four month period and maybe 2010, I want to say, where I probably went through, like, four or five Xbox’s alone in that year. So just every few weeks, I'm going back to work and whatever manager’s on duty is just gonna laugh it off and just wonder what's going on. And then I would go home with a slightly different Xbox to try my luck again.

JAMES PARKINSON: At first, Microsoft wasn't alarmed by a few returned consoles, but the Red Ring of Death soon escalated to the point that it threatened to destroy the entire Xbox brand.

JAMES PARKINSON: I’m James Parkinson. From Lawson Media, this is Gameplay, stories about video games and the virtual worlds that power culture and community.

JAMES PARKINSON: There are two main elements in developing a video game console; engineering and design. And both of these would play some part in the Red Ring of Death problem on the Xbox 360. But I think it’s important to talk about how Microsoft reached that point in the first place. Because the way the 360 came to be was drastically different to the original Xbox.

ROBBIE BACH: The initial product vision was actually closer to a personal computer than it was to a video game console. You know, in a way, this was a reaction to Sony. Sony was starting to market the PlayStation 2 as a personal computer in your living room, which turned out to be, sort of, both irrelevant and not actually true, but it's the way they were starting to position things, and that's what got Microsoft's attention. So the original idea was actually to basically put what was under the hood, a PC, in the living room as a video game console.

JAMES PARKINSON: That’s Robbie Bach.

ROBBIE BACH: My name is Robbie Bach, I was Chief Xbox Officer. I worked at Microsoft for 22 and a half years, did a bunch of things before we started on Xbox. But at the end of 1999, I became one of the leaders of Xbox, and then early in 2000, became the leader of Xbox. And I worked on Xbox for the next 10 years, until I left Microsoft in 2010.

ROBBIE BACH: Ultimately, as we spec’d out what Xbox would become, it was sort of a hybrid; it had a hard drive, like a PC would have, it had a network connection, like a PC would have. But it didn't really have very much of Windows, to be truthful. And it certainly didn't have multi-tasking or have any of the other elements of a PC - there was no mouse, there was no keyboard, none of those types of things. And so it really went more towards being a general purpose - or a single purpose game console, than a general purpose PC.

JAMES PARKINSON: The idea for Microsoft to get into the console market started in 1999. Some people at Microsoft were already thinking about what role they could play in the games industry, outside of PC games, in an informal “garage mentality” kinda way. But a company retreat that spring would elevate those ideas into reality.

ROBBIE BACH: And one of the things they do an executive staff retreat, or they did at the time, was we'd have this thing called ‘open space’, where anybody could propose a topic. And so, you go around the room, and maybe 15 topics would get proposed. And then the facilitator would kind of group people to the top six or seven topics. And people would sort of walk with their feet and decide which topic they wanted to talk about, and then you would go off and discuss them. And one of the guys who worked for me, Rick Thompson proposed that Microsoft should create a video game console to compete with the PlayStation 2. And so that was one of the topics for people to choose from. Bill Gates, ultimately, he had proposed a topic himself, which didn't turn out to be popular enough to make the final cut. So he decided to come to our topic. And we literally went up into a hotel bedroom, and sat around the room on desk chairs, and some people on the beds and whatever, and talked about what Sony was proposing with the PlayStation 2, and how Microsoft could compete with that. From that meeting, Bill went and met with the, before mentioned, garage shop groups to try to see if he thought there was a technical plan. He came back in July of that year and told Steve Ballmer, “hey, I think there's a technical plan. I don't know if there was a business to be made here”. And Steve asked Rick Thompson and myself to go look at that. And Rick spent the next four or five months working for me, but he was basically doing most of the work, I was still occupied on other things. And in December of 1999 - December 21 1999 - the plan for Xbox was approved. And shortly thereafter, Rick left the company and I became Chief Xbox Officer.

JAMES PARKINSON: There weren’t too many companies in ‘99 that were equipped to compete with the likes of Sony in the console market. It was a bold commitment for Microsoft, both financially and in the development time frame.

ROBBIE BACH: The original Xbox had an 18 month development cycle, which is incredibly unnaturally short. And the way we made that short is we bought a lot of parts off the shelf. So they were expensive. And then we weren't able to cost-reduce them and that cost us a bunch of money.

ROBBIE BACH: From the first discussion with Bill to green-lit was nine months. There was sort of a two month period, you know, and at the end of that nine month period, we only had 20 people. So there's kind of a two month period where the team started to ramp up. We had another meeting with Bill and Steve, where they sort of re-approved the project, because it became clear that what they thought they approved, wasn't what we were going to do. We call that meeting, the ‘Valentine's Day Massacre' meeting, because it took place on Valentine's Day. And then subsequent to that, you know, we really started to scale the team. And we literally hired, you know, almost 2000 people over the next 18 months.

ROBBIE BACH: I would say for that first 18 months for the first console, I was, sort of, ringmaster of a really out of control circus. You know, we were going from nobody to a big team of several thousand people. We were going from, you know, a general PowerPoint plan to a plan that sort of had specs. Then we had to make decisions about components, we had to start allocating money and people and resources. So every day was a little bit more of a, “come into the office, firefight what was going on, try to make some decisions”. Hope that we could get the team sort of coordinated and working in a common direction. And then, you know, work 18 hours, go home, have some dinner, sleep a little bit, come back and start over again the next day. And I know that sounds dramatic, but it was, it was a super stressful period. And there was not a lot of orchestration, there was not a lot of structure. It was, you know, “what's on fire today? Let's go fix that and move on to the next thing”.

JAMES PARKINSON: For the first generation Xbox, Microsoft were playing catch-up with Sony. The PS2 launched in October of 2000, and the Xbox in November the following year. Considering their plan for the console, Microsoft knew going into it that the Xbox was a financial risk. It ended up costing much more than they projected, and taking a hit financially, in order to fast-track the console would ultimately prove unsustainable.

DEAN TAKAHASHI: They did invest, what would ultimately become an insane amount of money in the first generation of the Xbox. You know, they had to sell the box for something like $300 or so. But the costs for making each box, you know, were coming in at something like $480.

JAMES PARKINSON: This is Dean Takahashi, Lead Writer for GamesBeat at VentureBeat. He’s also the author of two books on the Xbox and Xbox 360, respectfully.

DEAN TAKAHASHI: Well, it turned out that they had a lot of startup costs for the original Xbox. They lost about, something close to $4 billion over four years. It was like wrapping $100 bills around every console that they shipped out the door. And over the four years, you know, they realised, you know, we can't really continue this, we don't want to do this for eight years. Because we're just gonna lose eight billion dollars this way. And so, you know, they did sell - they had some hit games, like the original Halo. But the magnitude of the investment in the hardware and marketing and other things, was so big. And they needed to have a transition to the next generation, that would reset everything on costs. And so that was behind their decision to come up with the Xbox 360.

ROBBIE BACH: You know we said, “well, Xbox is gonna be more expensive to produce than PlayStation 2, because we've got the hard drive. So it's gonna be a little more expensive, we'll charge a little bit more, and that'll work out” - umm...really not so much. Basically, we had to price-match PlayStation 2 all the way down to $179. So we were losing, at most times, we were losing somewhere between $50 and $75 every time we sold a unit. So pricing matters, cost matters. You have to be able to have a powerful console, but you have to be able to produce it at a price people are willing to pay. And you have to be able to be price competitive with what other people are doing. And that's why there's this giant game of chicken every time there's a console change, because people are trying to figure out, “okay, what's the price point? How powerful is going to be? How do I have the same power they have without overspending? If I spend too much money and then don't succeed in getting the right chipset, I'm screwed”, you know, there's all this kind of complexity around that. And you have to get that right.

JAMES PARKINSON: Having lost several billion dollars on the first Xbox, Microsoft were determined to avoid a repeat. They also knew they had to get the jump on Sony next time around, and that actually started not long after the first-gen Xbox launched.

ROBBIE BACH: You know, as these things go, it started right away. Like literally, we launched in Europe in the spring of 2002, and the development team was asking us what we wanted to do for the next generation right away. So you did have this kind of, “hey, this train doesn't stop mentality”. So for the next generation, we knew we wanted to custom-make more of the parts, do more of it ourselves, and be able to cost-reduce it and design for cost reduction. And the only way to do that is to start early. And the first thing you have to start with is the silicon chip. Again, because that works in explicit time cycles, and you got to get on, figure out which wave you're gonna be on, what size, what performance, what cost structure you want to get on. And you have to make that decision four years in advance. With the original Xbox, we didn't have any choice, it was Nvidia's next chip and it was coming, and that's what we got. In Xbox 360, we were able to do a lot of custom work.

ROBBIE BACH: And so literally, when you're doing this design, you're making a bunch of bets. So okay, “we're gonna design on this chipset, in this form factor, in this time frame. And it's going to cost this much”. All of that as a guess, because the chipset hasn't been created yet. And the answers to those guesses actually matter. And if the chip slips, you know, suddenly you're late, now it’s something that's not as powerful, because somebody else has come out with the next generation. And oh, by the way, the cost is higher. So, you know, you have to make those bets. This is why, you know, if you think about what Apple's done, they've taken a huge chunk of their chip design in house, because they want to control it themselves. And so they get a lot of leverage. But those generations have to be planned four or five years ahead of time. And you can obviously do customisation and permutations along the way. But the core silicon layout gets done very early.

JAMES PARKINSON: With the first Xbox, as Robbie mentioned, the hard drive and other components were decided on first, with many of those being off-the-shelf parts. The shell of the console was very much an afterthought. For the 360, it was far more integrated.

ROBBIE BACH: We had design groups working on, you know, what the console would look like from the outside. And engineering teams looking at what the console would look like on the inside. And then the question was, “okay, we have to, how do we make those two things work so that the console works? Can we make it work inside the enclosure? Where does the power supply go?”. And all those kinds of questions. And so, in most respects, Xbox 360 was way more intentional and way more coordinated than the original Xbox was - certainly on, in terms of how the what's inside was integrated with what's outside.

JAMES PARKINSON: There are always risks in developing any hardware, but Microsoft were far better positioned for their second generation. Their strategy to get ahead of Sony paid off, at least initially, and the Xbox 360 had almost 12 months of runway on their rivals - a complete role reversal from the previous generation. The lineup of launch titles wasn’t the strongest, but there was plenty in the pipeline for gamers to be excited about. But less than a year into the console’s life, Microsoft hit a bump in the road.

ROBBIE BACH: When this first started it was, “oh, we have some reports of some problems, and we're exploring and investigating it”. And you know, your natural thought when you have something that's been working for five, six, seven months pretty successfully. And, you know, your son's been using it for five or six months very successfully, your natural thought and the natural thought of any, frankly, any consumer electronics company is, “okay, we have a manufacturing defect problem, we have a parts problem”, you know, “we have a component issue. One of our manufacturers isn't doing something right”, whatever it is. You go start investigating production lots and the subcontractors for various parts, and are they living up to the spec? And a whole bunch of other things. Because your natural assumption is you got some bad things that have gotten through QA and you have some bad units out there, and you need to isolate the source of the problem and eliminate it. Your first thought isn't, “oh, gosh, this could mean a billion dollar write-off”. That wasn't your first thought. It's not until the engineering team comes back to you two or three times and says, “it's not a manufacturing issue, it's not a components issue, we haven't found a design issue, we don't think it's a production capabilities issue. But it's clearly an issue, and it's getting bigger and bigger”. That's when you start to say, “oh, okay, now this is a real problem”.

JAMES PARKINSON: After the break, how Microsoft faced the Red Ring of Death, and saved Xbox.

[AD BREAK]

JAMES PARKINSON: At some point during 2006, word began to spread about some problems people were having with their Xbox 360 consoles. Internet culture wasn’t as mature as it is now - social media was barely a thing. But there was some talk on gaming forums, with people posting that their 360 was experiencing some issues, or had stopped working entirely. Here’s Dean Takahashi.

DEAN TAKAHASHI: It was very anecdotal at first, and Microsoft said, you know, to the press at that time, what they believe to be true, which was, “Oh, it's just a small number of machines”, you know, “most people are having a great time out there”, you know, “we're selling millions of these things, and a handful of people are having problems”, like, it's not - the return rates, or the failure rates on these machines are, you know, within the acceptable limits of what happens with every consumer product launch, right? They just didn't have this, you know, a whole body of people starting to complain. You know, fortunately, back in 2005, the internet isn't what it is today. Right. So we would have heard the complaints much more loudly today. But it seemed like this was under control.

JAMES PARKINSON: The general consensus from gamers at the time was that the Xbox 360 was overheating, which saw some bizarre and sometimes dangerous home solutions flying around the internet. Things like wrapping a wet towel around your console to keep it cool. Here’s Xbox fan, Tim David again.

TIM DAVID:  I remember, the two big ones that jumped out to me, we're as soon as people started to figure out that it was caused by an overheating issue - without necessarily tracking down what the specific cause was - a lot of people, they would talk about it the way people would talk about old cars, where they had to, like, make sure the airflow was regulated and try not to keep it next to anything else that was too hot. I remember there being a lot of internet superstitions around the idea that because the Xbox 360s power brick was so massive, and it gave off so much heat, that it was somehow giving off enough ambient heat to affect your 360 and caused that to overheat as well. So I remember, I definitely tried this myself. And I remember having a lot of friends both online and in person who would have these ridiculous setups where suddenly the actual power cable to their Xbox was a good two or three feet away on a different outlet, just to make sure it somehow wasn't accidentally microwaving your console. So it was a lot of that, it was a lot of trying to keep the fans from getting obstructed or to keep it in sort of a more open space. And I think - I'm not confident to say that ever actually prevented the issue for anyone, I'm not going to say I have the most comprehensive understanding of why the red ring was happening in this way. But I have to assume a lot of this was just sort of like luck and timing for a lot of people, where, “hey, yeah, I did move my power brick away from my Xbox, and I haven't gotten it yet”, even though we were bound to one way or another.

JAMES PARKINSON: The 360’s failure problems would eventually be dubbed the Red Ring of Death, and there was certainly a sense of anxiety amongst gamers. I remember feeling this myself at the time, and my own 360 eventually failed after a couple of years. The problem wasn’t confined to internet forums either. As it slowly gained momentum over many weeks, Tim recalls people coming into the games store where he worked to complain about their consoles.

TIM DAVID: At first, it was definitely a lot more of that kind of internet-based confusion, you know, it would be a lot of people coming in saying, “hey, look, I literally just bought this thing and the power buttons doing something weird”. And we’d kinda have to talk them through the steps of contacting Microsoft or returning it under warranty. Because for a long time, at Gamestop, or EB Games, if you had an Xbox that was doing the Red Ring of Death, we still couldn't do anything to take care of it, unless you had already purchased, like, the store warranty through us. So it was a lot of answering people's questions about what they think it might be. It was a lot of fielding angry customers who didn't know why they could only go back to Microsoft to get the problem taken care of.

JAMES PARKINSON: Microsoft initially treated the problem like any typical customer complaint and would deal with it on a case-by-case basis. But as former Microsoft Executive Peter Moore recounts, they were still struggling to work out what was actually happening. Here he is speaking with Ryan McCaffrey on IGN’s Podcast Unlocked in 2017.

Peter Moore: This was a thing that we actually couldn’t figure out what was going on. For those who don’t remember, what happened when the box bricked and died, you’d get actually three rings if I remember...

Ryan McCaffrey: Three red lights.

Peter Moore: And that was fine and we knew we had a problem. And I always remember going to Robbie Bach, my boss, and saying, “I think we could have a billion dollar problem here”. As we started to do the analysis of what was going on, we were getting the defectors in. It was a challenging problem for our engineers - couldn’t quite figure out what it was. We knew it was heat related.

JAMES PARKINSON: So, what exactly was the hardware failure that caused the Red Ring of Death, and why was it such a difficult problem for Microsoft to decipher?

ROBBIE BACH: So this is the guts of what makes the Red Rings of Death so interesting. It's not - if you asked the engineering team today, I don't know exactly what they would say. At the time, they could not point to any one thing and say that was the problem. If that had been the case, we would have tried to fix it very quickly. There were people who said it was heat. I can promise you that the Xbox 360 was heat tested before we shipped it extensively. So it wasn't solely a heat problem. You know, there's people who said it was airflow. And so now you had people on the internet, you know, posting pictures of a fan blowing on their Xbox 360. And what I ultimately believe - and I'm not not an engineer, but I spent a lot of time working with the team that was working on this. You had what I would call an interaction of parts and tolerances problem. And, you know, the tolerances for the box to work correctly, permanently, without having red rings and death were just too tight. And you know, that's a layout perspective, that's a components perspective, it's a heat perspective, that comes from a number of different angles. And so based on the way the tolerances were, a certain number of units were going to fail, and you couldn't predict which ones. Because when you have narrow tolerances on something, you know, in manufacturing, something can pass the initial manufacturing test. And if it passes with narrow tolerances on a number of different parameters, it might fail in the future. And that's what we saw happening.

JAMES PARKINSON: Dean’s understanding of the hardware failure points to a combination of problems.

DEAN TAKAHASHI: The graphics chip had this problem of - it was designed to be as ambitious as it could be, and so it ran hot. The memory that went into the system was supposed to be as high-grade memory as they could make, and it wasn't always the case that that was true. And when these things work together, you know, they developed problems. Like, if you had a very high-end game, using this thing, it would make the graphics processor run hot. When you had different extremes of, like, heat and cold for these motherboards, they could expand and contract. They used this lead-free solder that was kind of brittle. And if it cracked, then that graphics chip could become loose in the socket. And that's when that kind of occasion, you saw the red ring of death. So those there is, sort of, like this collection of different problems about you know, the different quality of the different components that we're using, like say, the solder along with the ambitious design of, like, you know, let's make the most badass thing we could possibly make. Yeah, it's gonna run hot, but we can deal with that. And not not having sufficient airflow in the machine to cool that down. So all of these problems kind of compounded upon each other. And that's why there were the failure rates that were through the roof.

JAMES PARKINSON: In September of 2008, Dean published a detailed article for VentureBeat, documenting the inside story of the Xbox 360 defect. He spoke to an engineer at Microsoft who repeatedly warned his superiors that there was a problem with units coming off the production line. This was happening around August 2005, as manufacturing was ramping up prior to the console’s release.

DEAN TAKAHASHI: One engineer at least had said, "Stop the line, we have to stop the line, you have to stop making these things, you have to figure out what's wrong. And then we can start it up again". And you know, he was overruled. And people above, all the engineering ranks above said, “no, just keep making them”. And so then you're getting then different kinds of numbers that were coming back, which we discovered in, you know, from internal reports, that among the first 11.6 million consoles that shipped, 1.2 million had failed in the field in the hands of gamers, right? That's a 10.3% failure rate. The Consumer Electronics Association, at the time said, most acceptable failure rates for new products were around 2%. And in the long term, this trended downward, they improved the manufacturing to some degree, figured out what some problems were, so it was more like 6% or 7% in the long run. But they had these things piling up, like, Wistron was one of the manufacturers, and they had this warehouse that was just completely loaded with failed Xbox 360s that had the red ring of death. And they called it a bone pile, because they could use those failed consoles for replacement parts. And Microsoft didn't realise how much this was going to hurt their reputation.

JAMES PARKINSON: Remember, Microsoft were desperate to get ahead of Sony and Nintendo. As Dean reported, they had to hit their launch target and ignored the warning signs as a result, believing that they’d be able to work out the kinks as they went and any issues would be solved over time. But Microsoft completely underestimated the scope of the problem, even as the Red Ring of Death began to gain traction. But eventually, it became clear that they had to act. In a business review meeting with then Microsoft CEO Steve Ballmer, Robbie Bach and Peter Moore laid out their plan. Again, here’s Moore on IGN’s Podcast Unlocked.

Peter Moore: Here’s what we had to do; we need to FedEx an empty box to a customer that had a problem - they would call us up - with a FedEx return label, to send you a box. And then we would FedEx it back to them and fix it. Either keep you hard drive or send it to us.

Ryan McCaffrey: Yep.

Peter Moore: And I calculated with my finance team, Dennis Durkin, Doug Ralphs - Dennis now the CFO at Activision - $1.15 billion. I said, “If we don’t do this, this brand is dead”.

JAMES PARKINSON: A billion dollar write-off; the biggest in Microsoft history at the time, and the biggest in game console history. On the 5th of July 2007, Microsoft announced they were extending the warranty on Xbox 360 to 3 years to cover the hardware defect. Here’s how Robbie Bach remembers that meeting with Steve Ballmer.

ROBBIE BACH: Which was - actually had nothing to do with that with the Red Rings of Death, it was just a regular business review for my entire business. And of course, this issue came up and he said, you know, you gotta do what's right for the customer. You know, if it costs us money, that's what we're gonna have to do, the right thing for the customer. So that enabled us to explore options that were expensive, frankly. But they were the right thing to do for customers. And then the second meeting is a meeting that for reasons I don't recall why, happened to take place at my house in my basement. But the management team came over to my house, at seven o'clock in the evening. We went through the various options, and decided that the best thing for us to do was to extend the warranty to three years - from one to three years in the United States, and from two to three years and in Europe. And that in that timeframe, we would capture everybody who had an issue. And the operations team would put in place a process to repair/replace every Xbox that people wanted to ship back.

ROBBIE BACH: Effectively what it meant is we were running two businesses. So we had the business of selling Xbox’s and selling games - the regular business, I'll call it. And then we had a second business, which was the business of repairing and replacing consoles when people had a problem. And we literally had separate facilities. Pretty much the same team, I don't think we added a lot of headcount to deal with this problem. But separate facility, separate vendors, a separate logistics process, and when you called us or emailed us or sent a carrier pigeon to us, saying you had a problem with your console, there was a trigger that got flipped and a box got sent to your house and you put your console in it, you sent it back to us, and we paid freight both ways. We either repaired it or we replaced it, sent the console back to you and away you went. And it was a money-losing business, but it was a customer satisfaction business.

JAMES PARKINSON: Microsoft realised that their reputation was on the line and although it was an expensive fix, they could afford to take the hit and take care of Xbox customers.

ROBBIE BACH: The amazing thing is that Xbox 360 is the highest customer satisfaction product I ever worked on. And our customer satisfaction did not go down during Red Rings of Death. It's actually kind of crazy. But what it tells you is the following; people love the product. And because Microsoft was willing and able to fix this thing for as long as it took to fix it, people were pretty patient with us. They said, “hey, you're doing me right, I get it, you have a problem, you're doing me right, you're fixing it, I get a new console, and away I go. And it just reinforces the fact that, you know, taking care of your customers and doing the right thing for customers will always benefit you. And in this particular case, it kept customer satisfaction with the product actually quite high.

ROBBIE BACH: And so, you know, you really learn that the console itself is just a conduit. And yes, it's important that you get the architecture, right, so the developers can produce great games, and it's gotta be graphically rich, and la la la - all that stuff. It doesn't mean the technology is not important. But in the end of the day, the thing people buy is games. They love the games on Xbox 360.

JAMES PARKINSON: Having read many stories from gamers and spoken to people like Tim David, who still persisted with Xbox, despite going through multiple 360 consoles, I think this is absolutely true for the vast majority of customers. And despite the Red Ring of Death and the negative attention is rightly received, the Xbox 360 is considered a big success.

TIM DAVID: I had invested so much in, like, games for it. And because it was the most popular console, so many of my friends had one. So if I had any hope of playing video games with my friends, or trying the same video games they were, nine times out of 10, I would have to have an Xbox. But even above and beyond that, there were so many, like, downloadable games that were only available for the Xbox that I was super into. And I really liked a lot of the exclusives. Honestly, the reasons I had probably aren't much different than the way anybody sticks with anything else, right? Like, everybody has their friend that's like a serial iPad or MacBook buyer. I was the same way with Xbox’s, right? Like, I was already enough in that ecosystem. And I already liked enough of what they were doing, that when one broke it wasn't a question of, “well, maybe I'm just gonna play more PlayStation or Wii now”. It was more a matter of, “okay well, there's a few weeks I'm going to go without an Xbox again. But that's fine because when it comes back I'm still going to be a quote-unquote, “Xbox person”.

JAMES PARKINSON: While the customer service side of the business was going on, Microsoft engineers were working towards a permanent fix; analysing the defective units and redesigning the console to eliminate the problem altogether. This came in the form of a second model, the Xbox 360 Slim.

ROBBIE BACH: The way the engineering team fixed this problem is they basically started changing components and design elements inside the box, one by one. And the problem got better, but didn't go away. And so with each rev we did on the console, things got progressively better; failure rates went down, consumers were shipping back fewer consoles, things were getting better and better. But it wasn't until they fundamentally reshaped the console literally. When we shipped the angular version of Xbox 360, if you remember that, that is the version that made the problem 100% go away. But all the different permutations of the sloped, what I'll call the slope version, the one with the smooth, curved front, all had varying degrees of Red Rings of Death challenges associated with them. And they were different in each iteration of the box. But we never did get it to go away in that. And so you were constantly working to, you know, our focus was 100% on customers. How do we make sure customers are playing games on Xbox 360? They have a console problem, we have them send it back, we send them another one. We send them one that we think is better. And you know we had people who had problems with two or three of them, and we sent them two or three consoles, that each of which were better and better, until we got it right. And that focus on customers is what saved the business.

James Parkinson: Do you think if you didn't act, you know, as quickly as you did, and make the decisions you did, that things could have turned out quite a lot worse for the 360’s reputation?

ROBBIE BACH: Oh, no question. No question. I mean, it's hard for me to imagine what would have happened. You know, while I'd like to say we were smart to do what we did, I think it was easier than that. We did what we had to.

JAMES PARKINSON: A big thank you to Robbie Bach, Dean Takahashi and Tim David. Robbie’s book on the business and success of Xbox is called Xbox Revisited. And Dean Takahashi is the author of Opening the Xbox and Xbox 360 Uncloaked.

JAMES PARKINSON: You know, a couple of other factors that made Xbox so successful was Xbox Live and also the focus on being a real multimedia hub in your lounge room. These days of course, Microsoft has its Xbox Game Pass service - the so-called Netflix of gaming. Which shouldn’t be a huge surprise, when you consider that Microsoft were one of the first to bet on streaming. On the other side of the credits, we’ll jump into the format wars with Robbie Bach in Games Archive. That’s after this.

CREDITS

JAMES PARKINSON: Gameplay is a production of Lawson Media. This episode was written and produced by me, James Parkinson.

JAMES PARKINSON: The Gameplay theme was composed by Breakmaster Cylinder, our artwork is by Keegan Sanford, and other music, this episode, from Breakmaster Cylinder.

JAMES PARKINSON: If you’re enjoying the show, I’d love your support to help make Gameplay sustainable, so we can continue to bring you amazing stories from the world of games. You can become a member at gameplay.co/membership.

JAMES PARKINSON: You can follow the show at gameplaypodcast on Twitter, Instagram, Facebook and Twitch, and come and join our community on Discord. You’ll find that and much more on our website, gameplay.co.

GAMES ARCHIVE SEGMENT

JAMES PARKINSON: So, we’re living in the age of on-demand media. Whether that’s movies or games, format wars are kind of a thing of the past. But when the Xbox 360 was in development, the next iteration of physical discs was up in air. Sony went all-in on Blu-ray for the PlayStation 3, while Microsoft opted for HD DVD. Most people would consider Sony to be the ones who made the right call in this case, but Robbie Bach doesn’t see it that way.

ROBBIE BACH: I would argue we won that battle. And the reason I would argue we won that battle is because Sony picked Blu-ray, which was more expensive for them to put in the box and delayed the time when they could ship the box, because Blu-ray wasn't ready. And in our case, we chose to do a very nice, high quality progressive scan DVD player, which cost a lot less. And for people who wanted to play movies, was perfectly good. But we made our bet on streaming. And so you could phrase it as “Sony bet on Blu-ray, we bet on HD DVD, and they won”. I don't actually think that's what happened. What I think happened is Sony bet on Blu-ray, and we bet on streaming. And the fight over HD DVD was a sideshow, and to some degree, an intentional one.

JAMES PARKINSON: In July 2008, Microsoft partnered with Netflix to bring its streaming service to Xbox 360 for Xbox Live Gold members. It was the first game console to offer a movie streaming service of any kind.

ROBBIE BACH: And streaming ultimately won out. And certainly from a cost perspective, streaming was way better. And, you know, ultimately, the partnership with Netflix came to fruition. And, you know, the rest has changed the way people think about how movies are delivered. So, you know, Blu-ray, unfortunately, was sort of at the tail end of disc technology, as it turned out. You know, it hasn't been anything since Blu-ray, really. And so Sony placed a big bet on that, and you know, maybe they think it was a great bet. I would argue it cost them a lot.

JAMES PARKINSON: I’m James Parkinson. Thanks for listening.

References:

• Xbox 360 defects: an inside history of Microsoft’s video game console woes

• Podcast Unlocked 201: Xbox Bosses Past and Present Share Stories, Secrets

• How Five Xbox 360 Owners Kept Their Launch-Day Consoles Alive

• Microsoft extending 360 warranty to three years

• Opening the Xbox: Inside Microsoft's Plan to Unleash an Entertainment Revolution

• The Xbox 360 Uncloaked: The Real Story Behind Microsoft's Next Generation Video Game Console

• Xbox Revisited: A Game Plan for Corporate and Civic Renewal