Augmented reality tool shown to help surgeons remotely guide first responders in battlefield-like scenarios

In a smoke-filled room, a first responder receives instructions from a surgeon through an augmented reality headset to perform an operation. (Purdue University photo/Edgar Rojas Muñoz) Download image

WEST LAFAYETTE, Ind. — War zones can be too far from operating rooms, preventing soldiers from receiving the surgery they need fast enough.

A Purdue University-led study is the first to show medics successfully performing surgery in life-like simulations of these war zones by receiving guidance from surgeons through an augmented reality headset. The work is joint with Purdue's School of Industrial Engineering and the Department of Computer Science.

The headset transmits a recorded view of the operating site to the surgeon, who can then use a large-display touch screen to mark up the recording with drawings of how to complete the surgical procedure. Augmented reality helps the first responder see the surgeon’s annotated instructions directly on their view of the operating field.

Researchers at Purdue and the Indiana University School of Medicine have been developing this technology, called System for Telementoring with Augmented Reality (STAR), since 2015.

Operating rooms across the U.S. have already started using augmented reality telementoring to virtually bring in the expertise of other surgeons on how to use a new instrument or better perform a particular procedure.

But this technology hasn’t made it to “austere” settings, such as a battlefield or forest thousands of miles away from a hospital, where a first responder could be treating injuries far too complex for their level of expertise, said Juan Wachs, Purdue University’s James A. and Sharon M. Tompkins Rising Star Associate Professor of Industrial Engineering.

“Augmented reality telementoring doesn’t usually operate well in extreme scenarios. Too much smoke can prevent visual sensors from working, for example,” Wachs said.

Researchers from Purdue and the IU School of Medicine tested STAR with first responders and surgeons at the Naval Medical Center Portsmouth in Virginia. The work is published in NPJ Digital Medicine, a Nature Research Journal.

The study evaluated first responders using STAR to perform on a patient simulator a common procedure that opens up a blocked airway, called a cricothyroidotomy. Even the responders with no or little experience performing this procedure prior to the study successfully operated after receiving instructions from surgeons through STAR.

The simulations took place both indoors and outdoors, including smoke and sounds of gunshots, explosions and helicopters. The researchers found that first responders more successfully performed the cricothyroidotomies with STAR than with just hearing a surgeon’s voice for each of these scenarios. If smoke made the visualization too unreliable, the responders could still do the operation when STAR automatically switched to audio-only telementoring.

STAR would need more durable materials before it’s used on a real battlefield, Wachs said, but the software is now proven to work in extreme conditions.

Wachs has recently formed a company, AI-Medic, that is working on artificial intelligence solutions to address communication disruptions that might occur while using augmented reality telementoring systems in the field.

STAR is protected by patents filed through the Purdue Research Foundation Office of Technology Commercialization. This study was financially supported by Office of the Assistant Secretary of Defense for Health Affairs (Award No. W81XWH-14-1-0042) and the National Science Foundation (Grant DGE-1333468).

About Purdue University

Purdue University is a top public research institution developing practical solutions to today’s toughest challenges. Ranked the No. 6 Most Innovative University in the United States by U.S. News & World Report, Purdue delivers world-changing research and out-of-this-world discovery. Committed to hands-on and online, real-world learning, Purdue offers a transformative education to all. Committed to affordability and accessibility, Purdue has frozen tuition and most fees at 2012-13 levels, enabling more students than ever to graduate debt-free. See how Purdue never stops in the persistent pursuit of the next giant leap at https://purdue.edu/.

Writer, Media contact: Kayla Wiles, wiles5@purdue.edu (working remotely, but will provide immediate response)

Sources:

Juan Wachs, jpwachs@purdue.edu

Edgar Rojas-Muñoz, emuoz@purdue.edu

Journalists visiting campus: Journalists should follow Protect Purdue protocols and the following guidelines:

• Campus is open, but the number of people in spaces may be limited. We will be as accommodating as possible, but you may be asked to step out or report from another location.
• To enable access, particularly to campus buildings, we recommend you contact the Purdue News Service media contact listed on the release to let them know the nature of the visit and where you will be visiting. A News Service representative can facilitate safe access and may escort you on campus.
• Wear face masks inside any campus building. Wear face masks outdoors when social distancing of at least six feet is not possible.

ABSTRACT

Evaluation of an augmented reality platform for austere surgical telementoring: a randomized controlled crossover study in cricothyroidotomies

Edgar Rojas-Muñoz¹, Chengyuan Lin ², Natalia Sanchez-Tamayo¹, Maria Eugenia Cabrera³, Daniel Andersen ², Voicu Popescu²,

Juan Antonio Barragan¹, Ben Zarzaur⁴, Patrick Murphy⁴, Kathryn Anderson⁴, Thomas Douglas⁵, Clare Griffis⁵, Jessica McKee⁶,

Andrew W. Kirkpatrick^6,7,8 and Juan P. Wachs ^1,4

¹School of Industrial Engineering, Purdue University, West Lafayette, IN, USA. ²Department of Computer Science, Purdue University, West Lafayette, IN, USA.

³Paul G. Allen School of Computer Science and Engineering, University of Washington, Seattle, WA, USA.

⁴Department of Surgery, School of Medicine, Indiana University, Indianapolis, IN, USA.

⁵Naval Medical Center Portsmouth, Portsmouth, VA, USA.

⁶Department of Surgery, and the Regional Trauma Services, University of Calgary, Calgary, AB, Canada.

⁷Department of Critical Care Medicine, University of Calgary, Calgary, AB, Canada. ⁸Canadian Forces Medical Services, Ottawa, ON, Canada.

DOI: 10.1038/s41746-020-0284-9

Telementoring platforms can help transfer surgical expertise remotely. However, most telementoring platforms are not designed to assist in austere, pre-hospital settings. This paper evaluates the system for telementoring with augmented reality (STAR), a portable and self-contained telementoring platform based on an augmented reality head-mounted display (ARHMD). The system is designed to assist in austere scenarios: a stabilized first-person view of the operating field is sent to a remote expert, who creates surgical instructions that a local first responder wearing the ARHMD can visualize as three-dimensional models projected onto the patient’s body. Our hypothesis evaluated whether remote guidance with STAR could lead to performing a surgical procedure better, as opposed to remote audio-only guidance. Remote expert surgeons guided first responders through training cricothyroidotomies in a simulated austere scenario, and on-site surgeons evaluated the participants using standardized evaluation tools. The evaluation comprehended completion time and technique performance of specific cricothyroidotomy steps. The analyses were also performed considering the participants’ years of experience as first responders, and their experience performing cricothyroidotomies. A linear mixed model analysis showed that using STAR was associated with higher procedural and non-procedural scores, and overall better performance. Additionally, a binary logistic regression analysis showed that using STAR was associated to safer and more successful executions of cricothyroidotomies. This work demonstrates that remote mentors can use STAR to provide first responders with guidance and surgical knowledge, and represents a first step towards the adoption of ARHMDs to convey clinical expertise remotely in austere scenarios.