For all the astonishing things artificial intelligence can do, it has a particular blind spot that one University of Virginia researcher seeks to remedy.
It can’t recognize all shapes.
“Current machine-learning models lack the capability to analyze and quantify the shape of objects presented in images with complex structures and large variations, especially in the context of medical imaging,” said Miaomiao Zhang, an assistant professor in the UVA School of Engineering and Applied Science.
The models are biased toward “seeing” image textures and have limited ability to perceive an object’s global structures, said Zhang, who holds joint appointments in the Charles L. Brown Department of Electrical and Computer Engineering and the Department of Computer Science.
“Given an image of a cat where its skin is replaced with that of an elephant, existing neural networks are susceptible to being deceived and misidentifying the image as an elephant,” Zhang said. “In such cases, the model focuses primarily on local image details derived from intensities or textures rather than recognizing the global shape of the cat as intended.”
This matters to Zhang, who specializes in developing AI for image processing and analysis in medical applications, frequently in collaboration with clinicians at UVA Health.
The AI tool set, including machine learning and deep neural networks, is a critical aid for diagnosing progressive diseases such as heart failure and Alzheimer’s. If AI can’t reliably recognize and quantify the geometric shape changes of heart motions or brain structures from medical images — magnetic resonance imaging, for example — it can’t help doctors precisely detect abnormalities or subtle changes caused by diseases over time.
Zhang’s project, “Deep Neural Networks That Can See Shape From Images: Models, Algorithms and Applications,” aims to create a new family of AI tools that can analyze and quantify shape information from images and use this perspective to improve image analysis and machine learning in general.
“Achieving this goal is crucial to unlock the full potential of AI in medical imaging, enabling more accurate and reliable diagnoses and empowering health care professionals with enhanced insights for improved patient care,” she said.
Scott Acton, chair of the Charles L. Brown Department of Electrical and Computer Engineering, said Zhang’s work is an important contribution in the recent explosion of AI models for health care.
“This proliferation is due to the research community adding structure to an unstructured machine-learning process,” Acton said. “Professor Zhang’s groundbreaking idea comes from a similar spirit. She is adding the ability to see global structure in digital images. As a biomedical image analysis researcher, I can tell you that this advance will unleash new descriptive powers for health research.”
To help address AI’s shortcomings in shape analysis, the National Science Foundation has granted Zhang a prestigious CAREER Award, which funds projects for early-career faculty in recognition of the potential they have demonstrated for leadership in research and education. Zhang was also recently recognized with a National Institutes of Health Trailblazer Research Award and in 2014 earned a Young Scientist Award from the Medical Image Computing and Computer Assisted Intervention Society, in which she is active as a member and an area chair.
The Research
Deep neural networks — a specific machine-learning technique — are algorithms carefully developed to emulate human learning processes. You can think of the algorithms, or mathematical formulations, as the neurons that are learning to recognize, recall and draw meaning from patterns in the data.
Zhang is proposing a new family of deep neural networks with the power to analyze shapes: deep shape-based neural networks, or DSNNs. To achieve this requires a shift in how researchers teach the networks to see and analyze shapes — and it’s not unlike teaching a child new words.
To develop the algorithms, her team will start by teaching the deep networks to analyze deformed variants of simple geometric figures — animals and everyday objects — using toy data sets such as Google’s Quick, Draw!, and adding more complex images as the model gains intelligence. The team will incorporate thousands of MRI and computed tomography images from Zhang’s clinical colleagues at UVA Health and from research databases maintained by Brigham and Women’s Hospital in Boston.
The outcome of this research will have immediate, long-lasting and wide-ranging impacts in the real world beyond medical imaging. Over the five-year project, Ph.D. students in Zhang’s lab will ultimately produce software that can be adapted for myriad uses, such as the processing of satellite images to monitor environmental changes in rivers and other geological features.
The project’s multidisciplinary nature also provides opportunities to attract students and researchers from diverse backgrounds across engineering, mathematics and medicine — and that makes Zhang especially enthusiastic about the project’s educational outreach component, an important element of CAREER Awards.
While the research involves advanced math that engineering students might not have encountered, Zhang plans to integrate her models into UVA courses in ways that are accessible to undergraduate and graduate students. She’s also eager to engage local middle and high school students through workshops and summer research programs to draw more women and students from other groups underrepresented in STEM fields.
AI: A ‘Co-Pilot for Clinicians’
Zhang’s collaboration with UVA Health’s cardiac center provides a unique opportunity to record video of her newly invented deep shape-based neural networks technology being used in medical procedures — for example, during heart surgery to implant a pacemaker.
“I can show potential future researchers how we use all this cool technology to impact a patient’s life and get them excited about the work,” she said.
Image analysis through artificial intelligence is transforming health care, Zhang said, and there’s a critical need to train surgeons in AI basics. UVA has begun to address this, including by inviting Zhang to participate in seminars that provide future surgeon-scientists with essential education in AI technology while emphasizing the importance of its responsible use.
“It’s not in the far future that AI will become a highly effective co-pilot for clinicians to make decisions on diagnosis and treatment plans, even during surgery in real time,” Zhang said. “However, neurosurgeons or cardiac surgeons will need to understand the basic science behind the software they’re using. If the algorithm delivers a false result, they need to identify it.”
As AI stakeholders, surgeons and physicians are indispensable partners in the work Zhang and other UVA Engineering researchers do across multiple disciplines, she said, and there has to be a continuous feedback loop between engineering and medicine.
“Then you create an interconnected and mutually beneficial education system for successive generations of engineers and medical students,” Zhang said. “That’s also a piece that I’m really enthusiastic about contributing to through this CAREER proposal.”