A heartfelt Augmented Reality story by Joy Kogowa.
I’ll admit, I’ve played through a lot of video games since my childhood. My first games console was the Atari 2600 - it was simple and primitive but it lead me to develop a very strong love and passion for well crafted video games. From first person shooters, RPGs to RTS. I’ve played through many genres and titles but none have really moved me emotionally, with the exception of games like: “The Last of Us” , “The Unfinished Swan” and “What Remains of Edith Finch”. These games deeply resonated with me, bringing me to tears on more than one occasion which I find really impactful. Until this point in my career I never had the opportunity to work on a game, my work has been mainly centered around experimental web experiences.
Shortly after completing Love Lost at Jam3, Jason Legge, a senior producer at Jam3 asked me if I would be interested in working on a game called “East of the Rockies”. Without any hesitation and not even knowing the story I said yes!
Jason then went on to explain how he and Dirk Van Ginkel, our Creative Director at Jam3, wanted to create a storytelling piece to mark Canada’s 150th anniversary. The story would focus on Canada’s injustices as well as their achievements through a fictional story of someone’s past.
This person was Joy Kogowa, a first-generation Japanese Canadian poet and novelist who lived through the Japanese Internment camps during World War II. She was able to take these negative experiences and turn them into creative outputs with one example being writing Obasan.
Jason, who has been a fan of Joy’s writings since his university days, reached out to Joy and asked her if she would be willing to partner with Jam3 to create a storytelling experience. After explaining his idea more and showing her “What Remains of Edith Finch” she was on board!
Joy paired up with Walt Williams to produce an original fictional script for East of the Rockies. Once our team had the script in our hands it was time to roll up our sleeves and get to work!
My role for this project was Lead Game Developer. I had a small group of creative developers and technical artists to manage aside from working on the project myself. My main tasks would involve developing the app architecture, handling the interactions, camera direction and working on additional shader effects and providing support for UI where needed.
As with most projects I start on, the first month or so is usually spent planning and creating various prototypes that test ideas to backup our creative vision for the project. Since East of the Rockies would be primarily an Augmented Reality experience, AR was one of the first prototypes on the list.
ARKit, Apple’s Augmented Reality technology launched in Summer 2017 so by the time we started development on EOTR there was no official Unity AR plugin. Instead we relied on Vuforia, an SDK toolkit that provides an API for using AR on devices. It wasn’t ideal but it worked for the time being.
Our creative team started to work on moodboards and art direction depicting the various locations the user would encounter during their playthrough.
The concept art by Aaron Campbell inspired the whole team. It set the tone perfectly well for the game and it gave our development team enough ideas to plan out how we’d achieve this art direction in Unity.
The story centers around three characters: Yuki, Tomio and Tak. We planned early on in the development of the game to motion capture actors performing the necessary actions we’d need for the story. We used the Rokoko motion suit to capture these various animations. In hindsight it wasn’t the best. It was difficult for our team to edit the animations and some had to be recaptured in several recording sessions.
The world in which East of the Rockies takes place would be physically large in Augmented Reality. One of our early ideas was to mask out the surroundings so the user could focus on a particular area in the scene. We called this our SceneMask. One our of lead creative developers Fabio used a post processing mask to fade out the render based on the AR Anchor position in the physical world.
We planned to make the game also playable without AR. I developed a universal camera animation system which panned, scaled and zoomed the world in and out based on the timeline of the story. This allowed us to direct and focus the user’s attention to specific moments in the story.
The game features various user interactions like: drawing pictures, picking up objects, interacting with elements in the world. Some of these are optional and hidden but most require the user’s interaction to progress through the scene.
As the user progresses through each act they unlock collectables that reveal historical context. Jason and Dirks had planned early on to develop educational resources for schools. Adding unlockable collectables was another way we could help educate the player.
Two months before the project deadline Unity released AR Foundation, a package that unifies AR development across iOS and Android devices. Even though the package was early in development I was impressed with the simplicity of the API and the small overhead of the codebase. I was still unhappy with Vuforia‘s performance as it would occasionally throw errors and the tracking would occasionally drift from time to time. I created a small prototype to test the feasibility and impact this would have on the game and noticed a significant improvement overall to the frame rate. With the game being a 30 - 40 minute experience our aim was to make it as smooth and performant as possible. This change was most certainly for the better!
We first premiered East of the Rockies at the IDFA Festival in Amsterdam November 2019. This was the perfect opportunity for our team to gather feedback on how people would play and experience the game. With the exhibition lasting several days we opted to use iPad Pro’s to ensure the game would run smoothly throughout each day. Upon arriving back to the office we made several UX improvements based on the feedback we gathered.
We launched the game on the iOS app store in March 2019 shortly after earning us App of the Day. The game was received positively from various news articles and award ceremonies. On April 30th 2019 I had the pleasure of publicly speaking about the game development at FITC Toronto.
I will never forget the first time I met Joy at the Jam3 office. She was so humble and appreciative of the team’s efforts to turn her story into something memorable and educational. Her experiences as an interned Japanese Canadian and how she was discriminated against for her identity draws many parallels to how others are being treated during the present day. I believe educational experiences are one of the many ways we can help people understand and learn to be more empathetic with one another 🖤.