Machine Learning Helps Track Objects in Complex Scenes Over Time

Robert Cardillo, a career intelligence analyst and former director of the National Geospatial-Intelligence Agency, reflected often on the evolving role of imagery to identify and understand threat. The U-2 photography of Soviet missiles in Cuba in 1962 is a quintessential example of images used for intelligence. Cardillo spoke of a world that has moved from data scarcity to data abundance, from making the case with one perfect image to telling a story with a flood of images.

“Imagine a coach trying to understand the strategy of his opponents by watching every play made by every team in every game for three seasons – all in one single day. Because three more seasons will be coming tomorrow,” Cardillo said. “That’s what we ask our analysts to do – when we don’t augment them with automation. But with all this data – and dramatic improvements in computing power – we have a phenomenal opportunity to do and achieve even more.”

The need for automation has grown in the three years since Cardillo made the analogy at the Geospatial Intelligence Forum’s annual symposium. The National Geospatial-Intelligence Agency’s 2020 Tech Focus report calls for innovations to help geospatial analysts detect objects and changes in an image with little or no description. In addition to longitude and latitude, analysts must also factor in movement across time.

Two researchers in the University of Virginia’s Charles L. Brown Department of Electrical and Computer Engineering are applying machine learning and control theory to meet this need. Professors Scott Acton and Zongli Lin have earned a grant from the U.S. Army Research Office to tackle a long-standing challenge in intelligence, to keep eyes on an object that may change in appearance or may be temporarily hidden from view. Acton and Lin are developing a visual object tracking system that switches between automated and manual analysis to deliver the best results.

Acton draws an analogy to “Where’s Waldo.” The first task is to find Waldo in a big mural. “We take a picture of Waldo, and then use features such as color, size and shape to find the silhouette within that large canvas,” he said.

Acton and Lin are developing a filter that outlines an object of interest in the scene. The filter uses machine learning to extract features. Because the computer decides what features are most important, the analyst spends less time looking and more time assessing what the image portends.

Extending Acton’s analogy, the bigger challenge is anticipating where Waldo will show up next. What if Waldo gets bigger, turns away, or walks through a tunnel? Acton and Lin are developing algorithms to keep eyes on an object along its journey at intervals of 20 to 60 minutes.

Think of a truck traveling along a highway. It might look like other trucks on the road. When it travels through a long tunnel, it would be indistinguishable from those other trucks. The computer, however, is trained to recognize tiny features such as angle, size, color, reflectivity and texture to set it apart. The algorithms cue other automated processes or tip the analyst to a movement that requires direct attention.

Acton, director of the Virginia Image and Video Analysis laboratory, is an expert in biomedical image analysis. However, he benefitted from an Army Research Office Young Investigator award and credits his exposure to Army missions for guiding his energy toward image analysis and object tracking. “I’m pleased that this recent research award allows me to come full circle, from military to health applications and back again,” Acton said.

Lin, Ferman W. Perry Professor of electrical and computer engineering, has made fundamental contributions to control theory. This discipline deals with the response of dynamic systems to various inputs, and applications range from computer hard disk drives to smart buildings, and neural networks to unmanned aircraft. Lin applies machine learning to control design for systems  whose mathematical models are not known.

The Army Research Office grant supports Acton’s and Lin’s long-standing collaboration to broaden and deepen connections between control and image processing. Almost a decade ago, the Army Research Office funded Acton and Lin for the development of grid-based trackers that could track moving objects in real-time with limited computing capabilities.  

“Object tracking is a fundamental problem in control theory,” Lin said. The connection between tracking and control theory goes back to the need to point a camera at a moving target, he said. “I am thus thrilled to have this opportunity to work with an image processing expert like Scott to explore application of machine learning in image processing.”