VR Game Blends Emotion Tracking With Interactive Storytelling
What if a virtual reality game could react not only to what you do but also to how you genuinely feel? Researchers at Northeastern University are exploring that idea with Rekindle, an experimental VR experience which combines facial emotion tracking with interactive storytelling. Rather than treating emotions as background data, the project makes them part of the gameplay, inspiring players to adopt the perspective of an LGBTQ+ protagonist through an immersive, story-driven journey.
Developed by Interdisciplinary Design & Media PhD student Hector Fan with professors Mark Sivak and Casper Harteveld, Rekindle is set in a fictional dystopian future in which an authoritarian regime manipulates memories to erase sexual identities it considers unacceptable. Players take on the role of the protagonist and gradually recover lost memories, but progress depends on more than uncovering story fragments. Their feelings also shape the experience.
The first-person perspective begins inside a neon-lit virtual nightclub, where ghostlike figures drift across the dance floor, and a stranger welcomes the player at the bar. Although the setting is fictional, it is designed to evoke authentic emotional reactions. To capture those responses, the Samsung Galaxy Extended Reality (XR) headset monitors facial movements connected to emotional expression. For example, it can identify a genuine Duchenne smile, a smile that engages both the mouth and the muscles around the eyes, allowing the system to distinguish authentic happiness from a posed expression.
Emotion recognition already exists in virtual reality, but Rekindle uses it differently. Rather than simply recording how players feel, the game weaves those emotions into its storytelling. Facial expressions become an active gameplay mechanic that forms the narrative.
Players advance by collecting memory fragments scattered throughout the experience. In contrast to traditional gaming collectables, these fragments represent meaningful moments from the protagonist’s life and are intended to reflect experiences that resonate among many LGBTQ+ communities. Each recovered memory gradually reveals more of the character’s background while encouraging players to participate in the emotions connected to those events.
Some memories reference real historical tragedies. One sequence draws on the 2016 Pulse nightclub shooting in Orlando, where a gunman killed 49 people and injured 58 others during the venue’s Latin Night event, which was widely attended by members of the LGBTQ+ community. Rather than recreating the attack itself, Rekindle incorporates the event into its fictional narrative to show its emotional impact on the community.
Memories alone are not enough to move the story forward. At key moments, players must also respond emotionally to their experiences. Instead of relying exclusively on controller inputs or dialogue choices, the game asks players to express emotions through their facial expressions, making those reactions an essential part of progression.
This approach intends to strengthen the connection between the player and the protagonist. Rather than observing another character’s experiences from a distance, players actively participate in the story by expressing emotions that correspond with key moments in the narrative. The result is a more personal interaction in which affective involvement becomes part of the storytelling itself.
The ideas behind Rekindle were presented in a research paper at the Augmented World Expo, a conference focused on extended reality and spatial computing held in Long Beach, California, in June. In the paper, the researchers argue that affective reactions in many interactive experiences have traditionally served as passive inputs. Players react to characters and events on screen, but those reactions rarely influence how the story unfolds.
Rekindle takes a different approach by turning emotional expression into an active gameplay mechanic. Instead of merely recording reactions, the experience uses them to deepen player involvement and encourage stronger emotional investment in the emerging narrative. This design shifts emotion recognition from a background feature to a central part of the storytelling experience.
Several scenes are intended to evoke experiences with deep cultural and historical relevance for LGBTQ+ communities. One confrontation, for example, is designed to remind players of events such as the 1969 Stonewall Riots and more recent clashes in West Hollywood. Rather than recreating individual incidents, the game presents symbolic memories that connect its fictional world with real moments in LGBTQ+ history. The psychology behind Rekindle has also drawn interest from experts working in biometric research. Graham Page, Executive Vice-President of Commercial Research and Innovation at iMotions, noted that most interactive systems record what users do, such as speaking commands or making selections, but provide limited insight into how users actually feel during those interactions. Incorporating emotion recognition, he argued, could provide a fuller understanding of user behaviour by linking actions to affective responses.
To achieve this, the Samsung Galaxy XR headset analyses facial movements and classifies them into six basic emotions, including happiness, sadness and surprise, as well as 15 compound emotions derived from combining those basic emotions. By observing individual facial muscle activity, known as action units, such as raised eyebrows or narrowed eyes, the system can estimate the intensity of each recognised emotion. Every detected emotional state receives a score between zero and one, indicating how strongly that emotion is expressed at a particular moment.
Although the technology measures facial expressions with considerable precision, affective reactions are rarely universal. Players bring their own backgrounds and personal experiences into the virtual world, meaning they may interpret scenes in very different ways. Moments involving acceptance, rejection or violence connected with sexual identity, for instance, may strike far more intensely with LGBTQ+ players than with those less familiar with those experiences.
The researchers see this variation as an important feature rather than a limitation. A player’s emotional response may not always match the emotional intent of a scene, but those differences emphasise how personal history shapes interpretation. In future versions of Rekindle, those individual responses could influence how the story develops rather than simply determine whether a player progresses.
That possibility opens the door to more adaptive storytelling. A scene that evokes a positive emotional response from one player but discomfort in another could lead each person down a distinct narrative path, creating a more personalised experience while promoting reflection on their own reactions.
The same emotion-aware technology could also prove valuable beyond interactive entertainment. Because it measures how people respond rather than simply what they do, researchers believe it could support a wide range of applications across education, healthcare and public safety.
Harteveld pointed to his own experience wearing eye-tracking glasses during a paragliding flight as an example of how biometric technology can assess stress in real time. Future systems that combine eye tracking with emotion recognition could go a step further by detecting rising anxiety and giving timely guidance, much like an experienced instructor helping someone through a challenging task.
Similar systems could also assist people during emergencies by recognising intensified stress and delivering calm, step-by-step instructions when clear guidance is most needed.
Rekindle demonstrates how emotion recognition might become far more than a tool for measuring player reactions. By making facial expressions a core part of gameplay, the project transforms affective involvement into an active storytelling mechanic, strengthening the connection between players and the protagonist.
While the research remains experimental, it points toward an era in which interactive experiences respond not only to what people do, but also to what they genuinely feel, an approach that could influence everything from immersive entertainment to healthcare, education and safety technologies.
