Congratulations to Dr. Tianshu Li for successfully defending her Ph.D. dissertation!! Tianshu joined the MOBLab in 2017 and has explored a variety of strategies for evaluating infrastructure systems ranging from visual recognition to natural language process. We are thrilled to see her defend her dissertation successfully. Tianshu's work has created novel approaches to using passive datasets to extract critical information on the performance of infrastructure systems. Looking forward to seeing more of her work published and shared with the academic community. A special thanks to her committee: Professor Arsalan Heydarian (ESE), Professor Michael Porter (ESE), Professor Vicente Ordóñez Román (CS), and Professor David Lattanzi (GMU-CEIE) for their contributions to her work.
Title: Mining Domain Knowledge from Unstructured Multi-modal Data for Smart Bridge Infrastructure Management
Abstract:
Improving the efficiency of the bridge infrastructure management in the face of critical preservation challenges calls for the integration of automation concepts through the process of inspection, condition assessment, and deterioration prediction. The current experience-driven condition rating process requires extensive effort in training and quality control to ensure the consistency of the assigned ratings. Additionally, the lack of details in the currently available bridge condition database limits the performance of the data-driven models that extract knowledge from past experience to guide decision-making in future maintenance. Meanwhile, the inspection reports generated through the current infrastructure management practices only serve as records of activities, leaving the condition details and domain expertise buried in the reports without being fully exploited for further analysis. To that end, this study identifies visual and textual data from bridge inspection reports as an untapped resource of bridge condition information and mines domain knowledge from a large number of historical inspection reports for automatic condition rating and information extraction. First, to improve the accuracy and consistency of bridge condition rating, a data-driven framework was developed as a supportive tool for two applications: automated condition recommendation and real-time quality control. A hierarchical architecture employing recurrent neural network encoders with an attention mechanism was developed using a collection of reports from the Virginia Department of Transportation. Next, to fully exploit the multi-modal data from bridge inspection reports, a fusion approach is proposed for automated bridge condition rating using the visual and textual data from bridge inspection reports. This study further investigated the uncertainty of rating predictions under random disturbance and showed that and referring the uncertain predictions to human investigations can further improve the rating performance. Lastly, an information extraction (IE) framework is developed to extract bridge conditions from the inspection reports at a high level of detail. Results of this study show that the proposed method can be used to extract and create a condition information database that can further assist in developing data-driven bridge management and condition forecasting models, as well as automated bridge inspection systems. This dissertation is a collection of three manuscripts that describes the aforementioned research works. Through the presented research outcomes, this dissertation highlights the value of unstructured bridge inspection documentation in supporting automated condition assessment and information extraction for the smart bridge infrastructure system.