Google Researchers Have Developed an Augmented Reality Microscope for Detecting Cancer

Augmented reality might not be able to cure cancer (yet), but when combined with a machine learning algorithm, it can help doctors diagnose the disease.

Researchers at Google have developed an augmented reality microscope (ARM) that takes real-time data from a neural network trained to detect cancerous cells and displays it in the field of view of the pathologist viewing the images.

Google Researchers Have Developed an Augmented Reality Microscope for Detecting Cancer — *Image by Google/YouTube*

The prototype, which is configured to identify breast and prostate cancer cells, does not require hospitals or labs to replace their existing microscopes, as existing compound light microscopes can be retrofitted with a digital camera for feeding images to the machine learning algorithm and a custom AR display for overlaying the digitized results onto the user's analog view.

The findings were presented on Monday at the Annual Meeting of the American Association for Cancer Research.

Don't Miss: Now the HoloLens Can Measure Your Heart Rate by Scanning Your Face

"We believe that the ARM has potential for a large impact on global health, particularly for the diagnosis of infectious diseases, including tuberculosis and malaria, in developing countries," wrote Martin Stumpe, the effort's technical lead, and Craig Mermel, a product manager on the Google Brain Team, via a post on the Google Research blog. "Furthermore, even in hospitals that will adopt a digital pathology workflow in the near future, ARM could be used in combination with the digital workflow where scanners still face major challenges or where rapid turnaround is required."

Previous research published by the team on the machine learning approach showed that the algorithm achieved a score of 89% in detecting tumors, whereas a trained pathologist without time constraints scored 73%.

Increasingly, machine learning and augmented reality are proving to be a perfect match of edge technologies. Google recently applied the technologies to enable YouTube creators to able to change video backgrounds in real time without depth sensors. And Facebook researchers have used the technologies to identify body poses for potential full-body AR effects.

But as entertaining as those experiences may be, they pale in comparison to the tangible benefits of improving the detection of cancer.