Augment Your Reality

JAMES PARKINSON: If you were to define calendar years by game releases, then 2016 was undoubtedly the year of Pokémon GO.

News Broadcast: “Pokémon go has only been out for a few days but it’s already got people following their smartphones to the most random places.”

News Broadcast: “This is just blowing up! Everyone I know is talking about this.”

News Broadcast: "This is huge, and here’s why - Nintendo has finally figured out how to augment reality. You pull out your camera and this creature appears in the real world and it superimposes that image. And so there’s this merging of real life and virtual life, and the result of that is that Nintendo right now is worth $7 billion dollars more than it was last week."

JAMES PARKINSON: The hit mobile game officially launched on July 6 and the sheer popularity of the Pokémon franchise saw it instantly rise to the top of app store charts. There was so much buzz about the game that its international roll-out was actually paused just two days after it went live because of server issues. But those temporary problems couldn’t stand in the way of Pokékon GO’s success. The game has been downloaded more than one billion times, with millions of monthly active users, ensuring it made billions of dollars in revenue.  

Pokémon Trailer: “The world of Pokémon is all around you. Go explore, go discover, go collect. Pokémon GO.”

JAMES PARKINSON: Pokémon GO was a defining moment in games for all of these reasons, but also because it really brought augmented reality games into the spotlight.

CHRIS MACKENZIE: It's a great evolution of, you know, Niantic’s, work in getting people excited about the idea of, you know, “take your device and walk around”. I think it's super important, you know? A lot of augmented reality demonstrations, you know, they have tried to experiment with the idea of introducing things in your world, but they really do have to sell that over an experience of just, “sit down at a PC screen, and enjoy a virtual world that you can kind of project yourself into”. So I think that one of the reasons that I see Pokémon GO was really important AR experience is they sold the cohesive experience of being a Pokémon trainer, going places, walking outside your house was core to the experience, and people were guided into that very early. I think they also had a good level of customisation as how they could play. You know, some people didn't use augmented reality at all. For others, it's a really important part of the game.

JAMES PARKINSON: That’s Chris Mackenzie from Opaque Media, who specialise in Augmented and Virtual Reality experiences. Augmented Reality, or AR, is sometimes the less understood technology, but if you’ve seen or used Instagram filters - the kind that overlay dog ears or a funny hat onto your head, that’s a simple form of AR.

CHRIS MACKENZIE: It's a very interesting term because, unfortunately, people have different ideas about what augmented reality is. In short, augmented reality is taking the real world, mediated through some computer device, and then injecting novel, different things into it, whether that's a Pokémon in Pokémon GO, whether that's furniture that IKEA can virtually place in your house to see if it will fit or not. Really, we're taking the real, and we're superimposing with the digital, in a way that you can see as if it's actually there in the room with you.

JAMES PARKINSON: While Pokémon GO wasn’t the first game to use AR, nor the first augmented reality game from its developer Niantic, it showed the potential of the technology to create new gaming experiences for a broad audience.

JAMES PARKINSON: I’m James Parkinson. From Lawson Media, this is Gameplay, stories about video games and the virtual worlds that power culture and community.

JAMES PARKINSON: In the discussion around new technology and games, virtual reality gets a lot of the attention. And while there are some impressive developments in the space, there are still lots of barriers. VR products are expensive and not everyone can stay in virtual reality for long, before they start feeling nauseous. One day, these barriers will come down, but what fascinates me about augmented reality is the fact it’s far more accessible, with practically every modern smartphone being capable of delivering an AR experience. Here’s Chris Mackenzie.

CHRIS MACKENZIE: The change we've seen in the augmented reality industry has really been people experimenting with new platforms, especially mobile. That’s really important for two really different reasons. The first is, we've seen augmented reality devices, which fundamentally, the most important thing is a camera. We can't understand the real world without having a camera, or some sort of analogous device, to get the information about what the real world into the computer, right? So we've seen better and better camera technology come along, and it's gotten smaller, and it's gone into our pockets. And the kind of second part of that loop is, we've got better and better graphics processing technology, which allows us to do two other things. We can run algorithms to understand the world that the camera is looking at, which is really important. You have to really, really quickly scan and understand where you are in a space, depending on what kind of AR experience you're making. But also, you know, where have you moved to? What's in the room? Where is it safe to draw things and where should you actually not draw things, because they might superimpose something that’s right in front of you, which would break that illusion of augmented reality. And the second thing we need to do is we need to draw, hopefully very nice looking graphics. And looking back, even, you know, at the technology and the platforms we had in 2010, we definitely had the ability to do all of these, but it wasn't really distributed in a way, which was very accessible to many people.

JAMES PARKINSON: The technology of augmented reality dates back to the early 1990s, and it was a very rudimentary period in its development. ARQuake is thought to be one of the very first AR games. It was an academic project out of the Wearable Computer Lab at the University of South Australia, which began in 1998. And it was a very simple adaptation of Quake, the first person shooter game, that overlaid some game elements, like your weapon and enemies, onto the environment around you. The player wore a custom built head-mounted display, with a camera, and a backpack that housed the computer and other hardware. The project actually remained in development until 2006, steadily improving, as the tech itself evolved.  

JAMES PARKINSON: By 2013, smartphones and camera technology had reached a level of maturity that put AR into the palm of your hand. Niantic, the studio behind Pokémon GO, was a startup that spun out of Google, and their first game was Ingress.

Ingress Promo Video: “Ingress is a giant game of capture-the-flag - where you play a video game, but in real life - Ingress is like Foursquare meets geocaching meets - giant game of Risk, where the board is like the surface of the Earth - there’s nothing really like it out there - you actually have to go out into the real world to play the game - Ingress is not a game - the line between reality and a game and a story get blurred - this is the kind of thing that everybody’s always wanted to do, like you bring video games into real life.”

JAMES PARKINSON: Achieving 20 million downloads and a passionate following, Ingress made a significant impact, but being a mashup of genres, and little complicated, it wasn't the most user-friendly. But when Pokémon GO landed three years later, millions of players were captivated by the game that encouraged them to get out into the world and engage with their surroundings. Naturally, it was a hit with Pokémon fans but it also reached a whole new group of players.

CHRIS MACKENZIE: I do think that AR has been very good at bringing people who wouldn't necessarily play games and introducing something that they, you know, it's calm, it's comfortable, it's something that they otherwise might not have picked up, and it enhances a part of their life that isn't necessarily analogous or replacing, kind of a leisurely “sit down and fire up the PC and play”. I think we have, as an industry, successfully plumbed a lot of that space. I do think there is more to go. I think that some of that might start to blend over into more of a social aspect. I think that people's relationships with their phones, and social media will continue to evolve as well, and maybe there'll be some interplay. It could also be that we see more people kind of taking Nintendo's route of a real-world additional thing that you can do to, you know, charge up the Pokéwalker, then plug that into another game. We could see more interplay between those systems.

JAMES PARKINSON: In 2019, Niantic released their next big AR title, Wizards Unite. Based on the Harry Potter franchise, its gameplay and mechanics are very similar to Pokémon GO, with the player casting spells to battle and capture beasts and other creatures from the Wizarding World.

Wizards Unite Trailer: “Defeat Dark magic, win wizarding challenges, and earn great rewards. Wizards Unite - download now”.

JAMES PARKINSON: While it was nowhere near the success of Pokémon GO, Wizards Unite still achieved 15 million downloads in its first month, bringing in $12 million in revenue for Niantic.

JAMES PARKINSON: In 2020, we saw a fresh take on the use of AR from Nintendo with the release of Mario Kart Live: Home Circuit. As a mixed reality game, Home Circuit uses real toy go-karts equipped with cameras, to superimpose the race track and in-game elements onto your environment via the Switch console.

Mario Kart Trailer: “Bring the kart racing action to life, right in your home. Mario Kart Live: Home Circuit.”

JAMES PARKINSON: It’s a classic Nintendo approach that blends toys and video games, with augmented reality tying the whole experience together. Nintendo seems committed to the AR space, having announced a partnership with Niantic to develop more of their properties for AR mobile games - including a game based on Pikmin, set for release in 2021.

JAMES PARKINSON: With huge franchises and companies experimenting with AR, it’s interesting to watch all of this play out and wonder where we might be with the technology in a few years time. Chris Mackenzie says he’s optimistic about the direction AR is headed.

CHRIS MACKENZIE: I'm really fascinated with the kinds of interactions that we can create, the kinds of user experiences we can create, which allow extension of your perception, I guess. I think that we can do a lot more than we're currently doing with the massive array of really cheap sensors, and really accessible network technology. I think that we might have the ability to let people you know, see in a different way. I think that augmented reality and the technologies that underpin it, you know, it might be that we find ways of overlaying things that we really care about, but can't really see, in ways that are fun to interact with as well. So I think those kinds of experiments about new experiences that are enabled by this, you know, much of augmented reality hardware and the software and the algorithms that are underpinning, it's going to create a lot of opportunity to explore, and that's something that I really love. You know, it's not just about solving the problem, but trying to understand what's going to be possible in the near future.

JAMES PARKINSON: Some of the most interesting innovations though, may not come from the big developers. Augmented reality is already being applied with a level of maturity from indie studios who are going beyond the scope of something playful, like a virtual pet, to create complex, story-driven experiences through the power of AR.

EMMA RAMSAY: We're making augmented reality games based on true crimes, true historical crimes, and you explore crime scenes, question witnesses and collect evidence to outsmart the killer and reclaim justice for the victim's family. And that’s all we need to say, really, and then people are just down for it [laughs].

JAMES PARKINSON: That’s coming up after the break.

[AD BREAK]

JAMES PARKINSON: Smart phones have created a whole new arm of the games industry, giving developers a platform to create new experiences. And AR technology has provided indie developers with another tool to play with.

EMMA RAMSAY: I just think there's enormous opportunity for augmented reality, and I feel like we're sort of only just scratching the surface.

JAMES PARKINSON: This is Emma Ramsay from Melbourne-based developer True Crime Games.

EMMA RAMSAY: We're making augmented reality games based on true crimes, true historical crimes, and you explore crime scenes, question witnesses and collect evidence to outsmart the killer and reclaim justice for the victim's family.

JAMES PARKINSON: Their first two games are set in Melbourne, Australia, as part of the True Crime Mysteries series. Misadventure in Little Lon debuted in 2019 and is set in 1910. The second title in the series is called Eastern Market Murder.

EMMA RAMSAY: In Eastern Market murder, it's based on a true crime from 1899, and it's geo located as well. So you can play from home or if you're in Melbourne, you can go to where the crimes actually happened, question witnesses, interact with them in a way, like, pouring tea for Mr. Friedman, or taking a letter from Mrs Grey, that you have to then pass on to someone else, that type of thing. And there's parts in the game where you actually use the vertical space as well. So the very final scene is at the location that used to be the Little Bourke Street watch house, and you use the brick wall there, where the criminal actually used to be. And you place the augmented reality scene up against that wall. So you're looking through the jail bars at the guy who committed the murder, and you're able to sort of take everything and all the evidence that you've learned throughout the game, to then confront him and bring him down, really. So it’s a very unusual use of augmented reality, but it is so perfect for our needs, in really creating that really gritty kind of experience and bringing back to life, these characters that were around 120 years ago, at the actual locations where they were as well.

JAMES PARKINSON: You can find a whole range of AR games on iOS and Android marketplaces these days, as well as apps that integrate augmented reality in some way. But True Crime Games are doing things a little differently.

EMMA RAMSAY: It's a fantastic experience and a really wonderful way to bring those stories from the past, back to life. A few people have described it as being like a futuristic window into the past and that kind of thing, which I just love. It is a fantastic way of getting to know a city that you've lived in, you know, possibly all your life, and it's a great way to guide people to places they might not usually go, and to not only see what's there now, but to easily imagine what was there over 100 years ago, and learn about the really fascinating things that used to be there and used to go on. To use quite an overused word, I think it's just, it really is the epitome of an immersive experience, when you're using your phone to uncover what used to be there 120 years ago and interact with the characters, collect evidence, and then confronting the killer at the end is super spine chilling, when you're doing it at the actual location where he was. And a lot of the script was based on letters that he wrote from jail, and the newspaper articles and the witness testimonies from the time. And their 3D models were created based on their actual photographs. So, you know, you're looking at the guy, exactly how he looked, and he's talking to you, in his own words, in the location where he was locked up 120 years ago. So yeah, I don't know if it gets much more immersive than that.

JAMES PARKINSON: And Emma says the reactions from players have been really positive.

EMMA RAMSAY: For the most part, their minds are completely blown, because I think it's just perhaps different from what they expected. Like, people who are used to playing games, and particularly people who've played Pokémon GO and that kind of thing before, they pick it up really easily, and it's kind of like second nature to them. But then we also have a rather large audience who don't play games, or they might just play casual games, kind of thing. But those people who are really interested in history and true crime and just love exploring Melbourne, they're seeing it is like a really fascinating way to explore the city and find out about what used to be there, and the types of people that lived there, and just like what life was like for them 120 years ago.

JAMES PARKINSON: True Crime Mysteries also shows the potential for AR games to be a great engaging tool for education, in a way that really pulls you into a story.

EMMA RAMSAY: There was a teacher who was teaching a crime and justice subject to Year 11 and 12s, and she was just absolutely blown away. Because, what's the usual method of teaching is via a book, and particularly history, it does tend to be quite dry, and to be able to take her students out and play this game, where the events happened, it's just a great opportunity and asset for educators to have in their pocket, I guess. And when you're looking at it, like compared to other games, technically, you can play it on the couch. Like, I've done that quite a lot during the testing, you just reposition the scene so they're practically right on top of you, so it is possible. But yes, it definitely helps people pay attention to the story when they need to physically and emotionally and mentally be involved while they're playing.

EMMA RAMSAY: It's really interesting that education and entertainment are so siloed and I just, I can't for the life of me understand why you wouldn't want to make education fun, and why you wouldn't want to, like, just learn a few cool things while you're playing games. You know, I don't really, I can't grasp or comprehend why the two are kept so separate. When there's like, benefits to blending them both in together, from both sides. Like, my nephew, he's 18, and he exclusively plays first person shooter games. But he came along and helped on our launch events. And he absolutely loved it. And at the end, we went out and sat down and had a meal together. And he was like, “so what's this phrenology thing all about?”, Like, “were they really into that? Why would they think that?”, and it just raised all these interesting things. And we had such a good chat after that. He was really excited and inspired. And it's like, “so dude, you just learnt a whole bunch of history!” [laughs].

JAMES PARKINSON: Unlike more traditional games, the advantage of AR is being able to create a more accessible experience for all kinds of players.

EMMA RAMSAY: To be honest, it's been one of our greatest challenges, because we want the game to still be really compelling and challenging for people who are used to playing games. But it also needs to be easy enough that someone who doesn't consider themselves into games, or has never played a game before in their life, that they're able to pick it up, and it's intuitive enough that it kind of guides them in how to place the scene and how to interact with the characters. And there's also a lot of previous games, that you just have to tap on things for things to happen. Whereas our game is more like, you're moving around the physical space and interacting with the space and the characters to progress the story. So it does sort of - is a bit mind bending for people who haven't used AR before, but it's also super effective in getting them interested and intrigued and wanting to play more games - not just in this space, but more broadly as well, because you're opening up the possibility space of what's fun and challenging. And you're also learning a bunch of stuff and going to new places and all of those good things as well.

JAMES PARKINSON: AR still has its drawbacks though. In the case of mobile games, they require you to hold your phone in front of your face. Until products like Microsoft Hololens become ubiquitous, it’s up to the game designer to smooth out those areas and make engaging with the technology a pleasant experience. For True Crime Mysteries, Emma says testing the game with players led to some important changes.

EMMA RAMSAY: The interface that we were using was just not conducive to playing a game one handed. So it was we had sort of more like a swiping motion that just made it quite difficult and a bit more taxing for the people who were playing. So we basically changed everything, so now you can play it one handed, and it's fully voice acted as well. So you can look at the characters and hold your phone up if you want to. But if you're just more into listening to the story, if you're more of like an audio inclined person, you can still just put your phone down and just listen for a while. So I think taking away that necessity to always be holding it up and just trying to make the UI as smooth and streamlined as possible, so it's easy for people to play, that's a way to overcome that challenge of it being kind of a bit more physically taxing to have to hold your phone up for an hour and a half.

JAMES PARKINSON: Popular games like Pokémon GO or Wizards Unite have the backing of huge franchises to demonstrate the power of augmented reality. But True Crime Games are showing that the technology is equally capable of mature and complex storytelling.

EMMA RAMSAY: We don't really know that many games that deal with true crimes at all, like, particularly, we don't know of any that blend augmented reality and true crime. There’s a couple that have popped up here and there since we launched ours a few years ago. But I'd love to be able to play more games like ours, really. Like, that's kind of why we made it in the first place, because it's just a game that we wanted to play and there wasn't really anything like that. So there seems to be heaps and heaps of really cute games that use AR, but maybe not as many things that you can really kind of get entrenched in this story of, and I think it's a fantastic medium for storytelling. And it's a great alternative to watching a movie, in a much more interactive way of uncovering those stories that are in every city, really.

JAMES PARKINSON: Emma says they want to explore historical crime stories in other cities for future titles, and are also thinking about how they can use AR to tell stories in other genres to reach an even wider audience.

EMMA RAMSAY: We're really looking forward to where our next games are going to go. We already feel like we've taken an enormous step from our first game compared to our second game. But yeah, I feel like the way that we're using it is really doing justice to the technology and sort of taking advantage and making the most of it.

JAMES PARKINSON: Many thanks to Chris Mackenzie and Emma Ramsay. Both games in the True Crime Mysteries series are out now on iOS and Android, and you can learn more at truecrimegames.com

CREDITS

JAMES PARKINSON: Gameplay is a production of Lawson Media. This episode was written and produced by me, James Parkinson.

JAMES PARKINSON: The Gameplay theme was composed by Breakmaster Cylinder, our artwork is by Keegan Sanford, and additional music from Epidemic Sound and Breakmaster Cylinder.

JAMES PARKINSON: You can follow the show on Twitter, Instagram and Facebook at ‘gameplaypodcast’. You can help us grow the show by sharing Gameplay on social media and with your friends. And come and join our community on Discord. You’ll find the link to join, plus episode transcripts and further reading on our website, gameplay.co. Thanks for listening.