Researchers at the Kavli Institute for Systems Neuroscience in Norway have developed an AI system that can predict gaze direction and eye movement from functional magnetic resonance imaging (fMRI) scans. The eye movements read by the AI system can help with the diagnosis of brain diseases and can identify patterns associated with thoughts, memories, and goals.
Eye movements provide insights into brain activity. For instance, when people recall memories, their eyes move in a similar pattern to how they did when the memory was formed. “Our viewing behavior is an expression of our thoughts, memories, and goals at any given moment in time,” said Matthias Nau, a postdoctoral researcher at the Kavli Institute for Systems Neuroscience.
Working with Markus Frey and Christian Doeller, he developed the DeepMReye eye tracker tool, which tracks eye movements from fMRI scans. The tool can be used to track eye movements without having to use cameras, works on existing datasets, and can track eye movements when the eyes are closed.
It is almost impossible to identify eye movement patterns in large datasets and associate them with cognitive processes or neural disorders; however, AI systems are good at identifying patterns and separating them from noise.
The researchers developed their AI tool to solve the problems associated with camera-based systems, which have prevented eye movement data from being used in MRI studies. Only 10% of MRI studies published in the top research journals have included eye tracking.
“Brain diseases manifest themselves as characteristic eye movement patterns and disturbances in viewing behavior,” said Nau. “Almost every cognitive or neural disorder, such as working memory deficits, amnesia, Parkinson’s disease, and Alzheimer’s disease will affect your viewing behavior.”
While it is generally agreed that eye tracking is a useful tool when investigating diseases and problems with cognition, the limitations of existing systems have prevented eye movement from being used by imaging research labs. Camera-based eye tracking is difficult to perform, time-consuming, and expensive. Camera-based systems cannot be used when the eyes are closed so it is not possible to study eye movements during sleep to identify sleep stages. Camera-based systems cannot be used to study eye movements in people who are congenitally blind, as the calibration process is dependent on vision and focusing eyes. These problems are overcome with DeepMReye.
Eye movement data is already present in existing fMRI scans, so the DeepMReye tool could be used to analyze fMRI scans from many thousands of individuals from 20 years of existing fMRI data, and could provide valuable insights from and forgotten old data.
DeepMReye – Source: Matthias Nau, Markus Frey and Christian Doeller. Kavli Institute for Systems Neuroscience
The AI tool is easy to use, does not require any equipment such as cameras, and does not even need to be used while the patient is undergoing an fMRI scan. It can be used post-hoc, long after the patient has returned home. The researchers’ open source software solution is available on GitHub along with user recommendations to make the tool as easy to use as possible. The researchers believe the tool can be used in a wide range of research and clinical settings.
You can read more about the DeepMReye tool in the paper – Magnetic resonance-based eye tracking using deep neural networks– which was recently published in Nature Neuroscience. DOI: 10.1038/s41593-021-00947-w