Augmented Reality Gives New Sight to Bioengineers, Doctors

More bioengineers are taking a serious look at how virtual and augmented reality tools, like Microsoft's HoloLens can expand their design capabilities, especially for medical imaging and computation, simulations, and implants and devices.

by Mark Crawford
March 12, 2018

Surgeons, bioengineers, and medical device manufacturers have long sought ways to improve their knowledge of human anatomy, ranging from using human cadavers and animal models to wax and plastic models. The goal, of course, is designing better treatments or medical devices/implants that simplify surgical procedures and improve patient outcomes, with minimal side effects.

As virtual reality (VR) and augmented reality (AR) tools continue to advance, more bioengineers are taking a serious look at how these technologies can expand their design capabilities, especially for medical imaging and computation, simulations, and implants and devices. VR/AR also enables shared or multiuser experiences that provide an enhanced level of education and discussion, whether it’s bioengineers working on a new invention or medical students studying anatomy in a classroom or laboratory. 

One of the most powerful AR/VR design tools for bioengineers is Microsoft HoloLens. This wearable holographic computer integrates 3D medical images with the physical elements of the user’s surrounding environment. The sophisticated optics and sensors in the HoloLens allow these images to be interactive, so they can be viewed by multiple users fitted with the same headsets. This way participants can walk around the holographic image (for example, a human form showing complex anatomy), looking for details they can use to improve a surgical procedure or optimize the design of an implantable device.

Amazing Clarity

The greatest benefit of HoloLens is the impressive visibility it provides to physicians and engineers. This allows them to better understand the systems they are studying, especially if they are working in torturous, tightly packed anatomy, such as the cardiovascular system or the brain.

Case Western Reserve University radiology professor Mark Griswold, a globally recognized radiology researcher, was immediately impressed after trying out HoloLens. Even though he had worked with datasets of brain MRIs for more than a decade, “I never fully understood their 3D structure until I saw them in HoloLens,” he says.

That, he says, is an improvement over dissecting cadavers to identify and study the body’s system. With HoloLens “you see everything truly in 3D,” he says. “You can take parts in and out. You can turn it around. You can see the blood pumping—the entire system.” HoloLens will be a key piece of technology in Case Western’s new health education campus when it opens in 2019, where students will study anatomy using virtual reality.

Bioinformatics researchers at Worcester Polytechnic Institute (WPI) are using HoloLens to visualize complex biological networks in 3D.

The 2D representations of these networks, such as disease systems, are so complex and dense that "they look like a big mess," says Dr. Dmitry Korkin, associate professor of computer science at WPI and director of the university's bioinformatics and computational biology program.

 Korkin is using HoloLens to develop new ways of "seeing" these complex networks.

“The topological complexity of such networks, and the abundance of the biomolecular and biomedical data, require a drastically new approach to integration and visualization of these data on the network structure,” Korkin says. “This has become a critical part of many engineer research areas that deal with designing complex engineering systems, from building the next-generation heart implant to mapping the molecular processes crucial in designing a new liquid biopsy technology. Mixed reality technology provides a natural way to visualize these complex data as virtual three-dimensional (3D) objects immersed in the real-world environment.”

Applications Abound

Stryker, a medical technology company, is using HoloLens to reconfigure operating rooms so they can better handle the needs of different types of surgeries. Medical technology firm Scopis has created a HoloLens-based mixed-reality interface for surgeons to use during surgery. CAE, a modeling and simulation firm, has also used HoloLens to create mixed-reality ultrasound simulations to help medical staff visualize complex organs using holograms. The University of Connecticut uses HoloLens as the main technology to optimize gait rehabilitation of amputees and stroke patients.

"Our goal is to merge different forms of information and look for patterns, for things that are unusual or abnormal,” Korkin says. “By allowing us to work with the complete set of data, which we simply cannot do in any other way because the information density is too high to comprehend, HoloLens makes this process more intuitive and more informative. We are all accustomed to dealing with a 3D world; this technology transforms complex biological systems into truly three-dimensional objects that we can interact with naturally and see in entirely new ways.”

Read about more key research technologies at