CASE STUDY

Mixed Reality Gaming

Creating the world’s first crime-solving video game for the HoloLens

How a last minute pivot shaped Mixed Reality gaming for HoloLens.

Role

Publishing Art Director at Xbox, led the Roadmap Lab

Deliverable

A new way to interact with holographic characters in the world’s first mixed reality crime-solving video game, Fragments.

Our Mission to Solve

Focus testing showed players struggled to see and connect with the 3D characters behind the wall‑mounted plinth, undercutting both usability and the platform’s mixed reality promise. For all of HoloLens’ magic, the experience was missing that spark of connection that makes it memorable.

(Footage from pre-rendered OLC Video)

RESEARCH INSIGHT

Mixed Reality gaming stands apart by blending fantastical gameplay with the player’s real-world environment, creating more memorable experiences through deeper emotional connections to both the story and its unique cast of characters.

Despite being in the 11th hour of development, we knew something was missing:

Breaking the Barrier

Research

Using the PRISM method, we dove Into the research phase, partnering across disciplines and with our partner studio, Asobo, to align on a set of non‑negotiables. These shared filters guided ideation, ensuring every discipline was driving toward the same solution.

Ideation

We leaned into HoloLens’ AI‑driven capabilities to craft a solution where our holographic crime‑solving team could interact seamlessly with real‑world objects, including walls, floors, and furniture, bringing the collaborative moments directly into the player’s space.

THE PRISM METHOD

Ideation Filters:

Characters respond to the Player

Holograms feel grounded in the world

Player feels like part of a team

Characters don’t get lost at harsh angles

Fits comfortably into the Field-of-View

Fulfills a futuristic fantasy

Working with our internal Publishing Concept Artist, we created a ton of ideas in parallel with Asobo Studio, generating a wide variety of technical solutions that we then stacked against the filter criteria from our Creative Process.

Leveraging AI‑driven world understanding, we built a solution that redefined player / character interaction

Introducing the Holo Call

Building on AI technology that mapped and understood living room surfaces, including furniture, Asobo utilized this tech, allowing characters to interact naturally with the player’s real-world spaces, improving UX and creating a deeper connection to the game’s cast of characters.

We led a secondary motion‑capture session to capture precise angles and human nuances, ensuring characters responded authentically to players while meeting the platform’s technical requirements.

Natural Interaction

We discovered that natural interactions between holographic characters did more than aid navigation and offset HoloLens’s limited field-of-view: they deepened immersion, grounded the fantasy, and played to the platform’s greatest strengths.
Eye Contact

After the technical hurdles, we authored behaviors into the characters to best capture their personalities. I worked collaboratively with Asobo to capture poses at various angles to maintain believable eye contact, no matter the height or position of the Player.
Improved Immersion

Seeing the game seamlessly juxtaposed with the player’s real world unlocked a new level of presence, made possible by spatial mapping and AI that allowed characters to understand the room and interact convincingly with walls, floors, and furniture.