Johns Hopkins Applied Physics Lab Receives $4 Million to Develop a Retinal Prosthesis

The Johns Hopkins University Applied Physics Laboratory (APL), in Laurel, Maryland, has received $4 million in funding from the Mann Fund to develop a next-generation retinal prosthesis system. The Mann Fund was created by philanthropist Alfred E. Mann 15 years ago to support the development of biomedical technologies.

This work will be conducted in close collaboration with Second Sight Medical Products, Inc., a Sylmar, California-based company that develops, manufactures and markets implantable visual prosthetics for blind individuals.

APL will be developing the next-generation glasses with embedded vision and eye tracking sensors. Using these sensors and onboard hardware, the system will identify potential obstacles, doorways, hallways, and household objects and their relative positions. This information will be distilled into a format that can be projected into the retinal prosthesis, bypassing the damaged rods and cones in the retina.

The components of this retinal prosthesis will enable the Laboratory’s broader vision of a semiautonomous controller for assistive robotic manipulators and remote devices, the Hybrid Augmented Reality Multimodal Operation Neural Integration Environment (HARMONIE).

HARMONIE is the latest spinoff effort from the Laboratory’s Revolutionizing Prosthetics Program. As part of that effort, researchers investigated a number of different control modalities for moving the modular prosthetic limb and were able to achieve impressive control. But they discovered that the user had to remain very focused on what they were doing, and there was a lot of cognitive burden.

It came about as a way to remove that cognitive burden from the user. The basic idea behind the system is to combine elements of computer vision (i.e., identification of objects and where they are), autonomous manipulation (how do I move the arm to the desired object’s position; what’s the best grasp to use) and a user interface (how do I tell what actions to perform on the object).

The current implementation uses off-the-shelf components such as the Microsoft Kinect for the vision system, but HARMONIE Project Manager Kapil Katyal said the ultimate goal is to integrate these technologies into a small form factor glasses type device.

The HARMONIE system has been deployed in clinical settings at Caltech and Johns Hopkins Medicine and is undergoing additional testing as part of an internally funded project, the Clinical Evaluation of Emerging Rehabilitative Technologies. It was also demonstrated in December 2013 at the DARPA Robotics Challenge Exposition, where RoboSally was able to use the technology to semiautonomously reach and grasp a fire extinguisher.

Katyal said they hope to have a prototype ready for market in 3.5 years.

“This system would enable a broad base of users to efficiently and effectively control dexterous manipulators like the MPL, and would also allow APL to develop innovative solutions in contemporary general-purpose Human Computer Interaction (HCI) technologies such as head-mounted displays, augmented reality, eye-tracking, and brain-computer interfaces,” he said. “Moreover, the synergies between the HARMONIE system and Second Sight’s next-generation retinal prosthesis provide an opportunity for the Laboratory to expand its footprint in neuroprosthetic research and development and enhance our reputation in this field.”

Michael McLoughlin, HARMONIE program manager and principal investigator for the Revolutionizing Prosthetics Program, said the HARMONIE project is synergistic with ongoing, independent research and development efforts at APL. “It’s an extension of what we’ve been trying to do with the prosthetics program from the beginning, and promises to address the needs of a whole new class of patients.”