How Augmented Reality is Helping with Surgical Precision

Augmented reality (otherwise known simply as AR) has to do with the art of superimposing computer-generated content over what we see in the real world. AR is possible through integrating digital information with our own real world environment occurring in real time. A majority of AR content can be viewed through the average smartphone.

VR is becoming more and more accessible (not to mention affordable) for new industries to adopt it. VR can be applied beyond the video game and virtual realm, branching off into industries to create a more educational and engaging experience in areas like self-help, marketing, medicine, dentistry, as well as nurse training.  

Augmented reality is proving to be particularly useful in medical education and imaging. Popular AR solutions that are already available include AccuVein, ARnatomy, VA-ST, and VIPAR.

Where Did Augmented Reality Come From?

AR was first developed, to a certain extent, by someone in the film industry — a cinematographer named Morton Heilig… in 1957. Heilig invented a contraption called the Sensorama, a device that delivered visuals, sounds, vibrations, and, yes, even smells to the user. 

This first iteration of AR was not computer controlled, however the Sensorama remains the first example of an individual making an attempt to add sensory data to a virtual experience.

In 1968, a man named Ivan Sutherland (an American computer scientist) invented a head-mounted display that could be used as a sort of window into a virtual world. However, this invention stumbled upon a primary problem in antiquated versions of VR and AR technology: these inventions were impractical for popular and wide use.

Then, in 1975 Myron Krueger (a computer artist) developed the very first virtual reality interface, an invention called Videoplace. Videoplace gave its users the ability to manipulate (and even interact with) virtual objects in real-time.

The very first properly functioning AR system is believed to be developed in 1992 at USAF Armstrong’s Research Lab — by Louis Rosenberg. Virtual Fixtures was a complex robotic system that Rosenberg designed in order to make up for lack of high-speed 3D graphics processing power that was available back in those times. Virtual Fixtures enabled the overlay of sensory information onto a common workspace, which was done in order to boost productivity.

Of course, more recently, advancements like ARToolkit (an Adobe Flash design tool), Google Glass, and Mocrosoft HoloLens have helped make augmented reality a household name… and this technology is finally ready to reach the masses!

AR and the Medical Industry

As far as the medical industry goes, AR is being used in conjunction with smart glasses that can superimpose real-time data sent from a dental scanner, which gives dental practitioners the ability to build more precise crowns and caps.

Nurses are now utilizing AR-enabled, tablet-based simulations that expose trainees to a variety of different patient scenarios. As a result, nurses are better able to connect with patients and are better prepared to handle daily work situations.

AR can also help enhance CT and MRI visualization by superimposing stereoscopic projections, which can be used over the course of an actual surgical procedure. This information helps medical professionals better perform surgeries that require extremely precise navigation, especially when it comes to a very particular organ. 

AR can also be used during pre-operative planning, enhancing accuracy when it comes to locating tumors and surrounding structures, especially in instances where the anatomy of the vascular or nervous system ends up complicating a tumor removal procedure.

AR and the VHA

The Veterans Health Administration (VHA) has partnered with Verizon, Medivis, and Microsoft in order to develop Project Convergence, which is an initiative to provide veterans with 5G medical campuses in the United States as well as sophisticated surgical visualization tools — campuses which will be the first of their kind.

The initiative itself is led by the VA’s National Center for Collaborative Healthcare Innovation. The way it works is by fusing together the following:

• Verizon’s 5G technology
• An FDA-approved use case in Medivis’ surgical augmented reality clinical visualization software (SurgicalAR)
• Microsoft’s HoloLens 2 headset with Azure cloud service

So what do all these awesome ingredients add up to? This initiative aims to improve healthcare delivery at VA facilities, and, ultimately, in the broader healthcare community at large.

More on Medivis

Medivis has developed a SurgicalAR platform that is truly groundbreaking, integrating augmented reality, artificial intelligence, as well as computer vision in order to help advance medical procedures like surgical planning.

SurgicalAR leverages the Microsoft HoloLens 2 AR headset, which provides a holographic visualization for physicians, allowing for far better precision, helping drive effective real-time decision making. SurgicalAR is being used both in and out of the operating room as we speak.

Neurosurgeon Osamah Choudhry and radiologist Christopher Morley are the pioneering physicians behind Medivis and SurgicalAR. The pair began laying their groundwork for SurgicalAR when they were still senior residents at NYU Medical Center. 

From the very beginning, these two doctors recognized the essential need for 3D technology and that this sort of technology could vastly improve surgical precision by providing surgeons with more comprehensive and immersive views of imaging. Today’s most commonly used imaging tools happen to be 2D currently.

Surgical teams have struggled in the past to understand critical 3D anatomy while scrolling through 2D magnetic resonance imaging (MRI) slices. Not only that, they’ve also struggled to mentally re-create what they’ve been able to see through a microscope.

The obvious question for Dr. Choudhry and his colleagues was: Why use 2D imaging when solving a 3D problem? 

This problem put Dr. Choudhry and Dr. Morley on the correct path that led to the development of SurgicalAR, not to mention the use of AR (as well as AI) to create extremely exact 3D holographic images, images that are much more helpful for surgeons who need to perform their jobs and procedures with the utmost precision.

Medivis has undergone a meteoric rise to prominence when it comes to the world of surgical visualization and that rise has partly been fueled by amazing partnerships with world-class technology companies — which includes Dell Technologies, NVIDIA, and other notable names.

“The way we work is that we take our imaging and we reconstruct it using our software technology,” Dr. Choudhry explained in a recent case study by Dell Technologies. “And because the reconstruction is very compute-intensive, we do it using a Dell Precision 7750 workstation with an NVIDIA Quadro RTX 5000 GPU. That system uses parallel processing capabilities that allow the holographic rendering to be created in real time.”

It’s Gotta Be the Glasses

Wait, we feel like we’re missing one important part of the SurgicalAR solution. Oh yeah, the Microsoft HoloLens 2 AR headset. Dell Precision workstation sends the data to the HoloLens, which helps the user visualize content in immersive 3D powered by the Verizon 5G network. 

But can AR be used beyond the operating room? With this new SurgicalAR solution, doctors will be better able to plan and prepare for surgeries of various types and complexities, not to mention view and orient themselves to the patient’s internal anatomy virtually. AR will only help the medical field create a more and more immersive learning environment for existing and training doctors, not to mention nurses and other professional clinicians.

AR is proving to be especially beneficial and important for VA hospitals, which provide a training ground for over 70% of doctors in the United States.

The SurgicalAR solution is scheduled to be deployed at VA sites in: 

• Palo Alto
• Portland
• Puget Sound
• Philadelphia
• Orlando

However, the larger vision of this incentive is to bring these sorts of surgical precision improvements powered by AR to VA facilities and other healthcare service providers across the country and ultimately the globe.

So, it’s clear that AR and VR are being used for far more purposes than simply creating an awesome video game or a convincing online/virtual experience. AR and VR have the capacity to touch and change a variety of industries and their applications are becoming more and more practical. Soon, even children will be able to use AR and VR technology with ease just as they do a smartphone.

The medical industry itself is fertile ground for exploring the true uses and benefits that humankind stands to gain from augmented reality technology, especially when it comes to complicated operations and procedures like surgeries. Now that AR is being applied by the masses, it will continue to change the way we interact with our world and each other.