Mixed Reality Social Prosthetic System

Abstract

A Johns Hopkins University Applied Physics Laboratory (APL) team conceived of and developed a first-of-its-kind mixed reality “social prosthetic” system aimed at improving emotion recognition training and performance by displaying information about nonverbal signals in a way that is easily interpretable by a user. Called IN:URfACE (for Investigating Non-verbals: Using xReality for the Augmented Consideration of Emotion), the proof-of-concept prototype system uses infrared sensors to measure facial movements, pupil size, blink rate, and gaze direction. These signals are synchronized in real time, registered in real space, and then overlaid on the face of an interaction partner, such as an interviewee, through a mixed reality headset. The result is dramatic accentuation of subtle changes in the face, including changes that people are not usually aware of, like pupil dilation or nostril flare. The ability to discern these changes has applications in fields such as law enforcement, intelligence collection, and health care. This article describes how the system works, the technical challenges and solutions in designing it, and possible areas of application.