UVA Engineering researchers and first responders join forces to devise cyber-physical system that supports emergency medical technicians who are busy helping others

UPDATE: In April, the team was awarded a new $1.1 million grant from National Institute of Standards and Technology's Innovation Accelerator Program to develop an expanded application of CognitiveEMS that gathers multimodal sensor data from the incident scenes and interacts with a team of responders through augmented reality devices.

George Stephens has been the chief of the North Garden Volunteer Fire Company for 20 years, and a firefighter even longer. He joined the company as a young man after he watched firemen struggling with inadequate tools and unreliable water resources try to salvage a house.

“Despite their heroic efforts, the house burned to the ground. It was the home of the custodian at my elementary school,” he said. “I realized I had just watched on the sidelines. That day I asked myself what I was doing to help and got involved.”

A lot has changed since he first volunteered. “Initially we only responded to fires, but over time we were responding to more car accidents,” he said. “We decided to get medical certifications because we wanted to do more for our residents than just hold their hand and comfort them until the ambulance arrived.”

Yearly calls have more than doubled since the early-1990s, from 300 to over 800 in a typical year. Today, close to two-thirds of distress calls are for medical emergencies or car crashes.

So, when a team of University of Virginia School of Engineering faculty approached Stephens to see if the company would collaborate on development of a new technology that could revolutionize first responders’ work, he didn’t hesitate.

Homa Alemzadeh, assistant professor in the Charles L. Brown Department of Electrical and Computer Engineering, is lead researcher on the new technology. Her collaborators are John A. Stankovic, BP America Professor of Computer Science and director of UVA Engineering’s Link Lab for cyber-physical systems, and Ronald D. Williams, associate professor of electrical and computer engineering.

  • Homa Alemzadeh

    Homa Alemzadeh, assistant professor in the Charles L. Brown Department of Electrical and Computer Engineering, is lead researcher on CognitiveEMS.

  • John A. Stankovic

    John A. Stankovic, BP America Professor of Computer Science and director of the Link Lab, is a researcher on CognitiveEMS.

  • Ronald D. Williams

    Ronald D. Williams, associate professor in the Charles L. Brown Department of Electrical and Computer Engineering, is a researcher on CognitiveEMS. Williams is also a 25-year veteran first responder and life member of the Charlottesville-Albemarle Rescue Squad.

They are developing a system which listens to conversations by first responders during emergencies and then offers helpful prompts to improve the responders' situational awareness and support them as they make decisions. The artificial intelligence technology is a cognitive assistant specific to emergency medical services, or EMS, and is called CognitiveEMS.

The responders of North Garden Volunteer Fire Company signed on to help with emergency scene simulations for data collection and for evaluating CognitiveEMS. Richmond Ambulance Authority joined the effort as well and provided emergency response transcripts and other data for the testing.

Stephens, who also happens to be a 42-year employee at Northrop Grumman, leads a team of electrical and mechanical engineers in marine navigation and guidance systems. He understands the value of collaboration.

“It’s hard to simulate a real-world environment in the lab and even harder to account for factors that come into play on the ground and in public-facing situations,” he said. “I am glad we can lend our front-line expertise to the project.”

The responders have been working with UVA on the prototype since the researchers earned the $1.1 million grant from the U.S. Department of Commerce’s National Institutes of Standards and Technology in 2017. The project was one of 33 funded under the Public Safety Innovation Accelerator Program, which is focused on key technology areas that have the potential to transform the future of public safety communications and operations.

Alemzadeh became principal investigator on the project just a few short months after joining the UVA Engineering Link Lab, where over 40 faculty and more than 200 graduate students combine efforts to develop and deploy cyber-physical systems. One of the Link Lab’s research focus areas is smart and connected health care.

“Link Lab promotes research that cuts across traditional boundaries and creates partnerships with the stakeholders who will benefit from the cyber-physical systems we are developing,” Stankovic said. “The CognitiveEMS project is a powerful example of how this approach leads to solutions that broadly benefit society.”

“I enjoy thinking about the real-world implications of research,” Alemzadeh said. “A device that assists responders while they are delivering medical care at the scene could reduce stress and cognitive overload and improve outcomes for both the patients and responders.”

Alemzadeh and her team shared their progress and a CognitiveEMS demonstration at the 2020 National Institute of Standards and Technology Annual Public Safety Broadband Stakeholder Meeting, held virtually in July.

  • Cognitive EMS Demo

    CognitiveEMS assists emergency responders by listening to their conversations at the scene and then offering helpful prompts to improve the responder’s situational awareness. (Demonstration image courtesy of CognitiveEMS research team)

  • Demo Only Sample CognitiveEMS Incident Report

    CognitiveEMS generates a real-time incident report, as seen in this de-identified example. (Demonstration image courtesy of CognitiveEMS research team)

CognitiveEMS listens to a first responder talking at the scene, gathering vitals, symptoms and facts; it then converts the audio to text. Trained through machine learning, CognitiveEMS extracts key semantic concepts from the text to pinpoint corresponding EMS protocols for the situation.

By modeling the important steps in the EMS treatment protocols and real-time monitoring of their execution by responders, CognitiveEMS continuously generates what it has determined to be the most helpful prompts—additional factors responders might consider as they make decisions about interventions. The system also takes into account the level of uncertainty in the extracted information from the scene and shows evidence and confidence scores for each prompt to build trust with the responders.

In developing CognitiveEMS, the researchers first had to create a domain-specific ontology of EMS concepts and terminology. They also had to build labeled datasets of simulated EMS audio transcripts of responders based on models of treatment protocols used by those responders.

The ontology and the behavioral models of EMS protocols connect the things responders do and under what conditions they do them—as identified by the words spoken at the scene—to the most relevant EMS protocols. The team has made these available to the research community through an online public repository.

Williams, in addition to being a researcher on the project, is a 25-year veteran first responder and life member of the Charlottesville-Albemarle Rescue Squad. He stresses that the CognitiveEMS prompts are not intended as guidance, but as input for responders to consider.

For example, an emergency medical technician might be responding to a call where the patient has difficulty breathing. Because fever could be a distinguishing factor between chronic disease or infection for that symptom, CognitiveEMS might prompt the responders by asking if they had considered taking a temperature.

“We were not looking to tell a responder what to do. We always want to leave the decision-making with the responders, so that they are always in control,” Williams said. “The objective is to reduce the cognitive load on the responders, helping them to focus on the highest priority tasks throughout the process.”

Another way CognitiveEMS reduces responders’ burden is through its automatic forms feature. During the incident, the system extracts timestamped information that is used to generate real-time incident reports that can be edited prior to submittal.

Williams understands why responders have indicated this feature is one of the most helpful.

“Documentation is difficult to do in the field when you are focusing on keeping your patient alive. The only practical means for filling out the forms is after the patient is in the hands of hospital staff,” Williams said. “I remember having to go down the hall to another room and fill out a form that documented everything you did, everything you saw, and the times these things happened. That’s just tough.”

George Stephens, chief of the North Garden Volunteer Fire Company

George Stephens, chief of the North Garden Volunteer Fire Company, is thrilled to be an active participant in developing an advanced tool like CognitiveEMS.

Stephens appreciates an additional advantage of the system’s automatic data capture.

“Time is irreplaceable in emergencies. If you can shorten the timeline to that first medical intervention by even ten minutes, then that’s a real game changer,” he said. “CognitiveEMS is continuously relaying what is happening with the patient in real time. Ambulance medics enter the scene with a much better understanding of the problem, so they can start life-saving interventions in transit to the hospital.”

Thinking back on the fire that led him to volunteer in the first place, Stephens is thrilled to be an active participant in developing such an advanced, next-generation tool. “As a rural fire department getting the opportunity to partner with UVA—by lending our services to help take tech into the future—well, that’s really neat,” he said.

Next steps on the project include fine-tuning the system for better human-cyber-physical system interaction that will transform CognitiveEMS into an at-the-scene agent. The researchers also aim to house the processing of CognitiveEMS locally on the wearable device, so that it is stand-alone and doesn’t need internet.

“In order for responders to rely on CognitiveEMS, they have to know that it will always be there,” Williams said. “We want to go the extra mile and ensure that a loss of communication does not take the system away from the responder.”

The UVA team will continue to test their system using realistic datasets and scenarios, during simulated training experiments, in collaboration with the emergency responders that have been pivotal in developing CognitiveEMS from the outset.

“Working with the responders who would use the technology in the field has been critical for translating research into actionable results,” Alemzadeh said. “We are grateful to the first responders who are helping us make CognitiveEMS more accurate, functional and reliable.”

In April, the team received a new $1.1 million grant from the National Institute of STANDARDS TECHNOLOGY PUBLIC SAFETY INNOVATION ACCELERATOR PROGRAM - AUGMENTED REALITY. Alemzadeh, Stankovic and Williams, are co-investigators on the grant. Funding will support the development of an expanded application of CongitiveEMS to more complex emergency scenarios and will include hands-free interactions with teams of responders through augmented reality devices.

Integral to the expanded CognitiveEMS, sensors will continuously gather incident data from all the responders as they are simultaneously working at the scene. By combining multiple streams of audio, video, and body motion data, the new application of CognitiveEMS can infer the multi-dimensional incident context, including patient conditions, scene situation, and responders’ actions, to improve situation assessment and support identification of the most appropriate responses to the situation as it unfolds in real-time.

The three-year project will leverage and expand on the machine learning, natural language processing, and behavioral modeling frameworks the team has already developed. It will also integrate those frameworks with new machine intelligence capabilities such as computer vision and sensor analytics to assess the situation and determine timely, customized suggestions and reminders.

Information will be provided to responders through hands-free wearable devices, like smart glasses, to increase interactivity between CognitiveEMS and the responders.