HARMONY: Driver State in Naturalistic Driving

Topics: #Naturalistic-Driving

The HARMONY study measured how drivers respond to real roads, using wearables, cameras and phones in naturalistic driving. Models linked stress and emotion, estimated from heart rate and other signals, to road objects, traffic and driving context.

As vehicles take on more of the driving, they need to recognize when the human driver is stressed or overloaded. Earlier naturalistic driving studies relied mainly on video, which shows what a driver does but not how the driver feels. HARMONY added physiological and environmental sensing to follow drivers' states over long periods of everyday driving.

Approach

Participants drove their own vehicles with a dash camera that recorded both the cabin and the road ahead, and wore a smartwatch running a lab-built app that logged heart rate and hand motion. The study also recorded ambient noise and light, GPS location and the music drivers played. Computer vision extracted facial expressions, gaze and head pose from the cabin video and identified road objects in the outside view. Statistical models, including Bayesian change-point detection and latent-variable state-space models, related these signals to what was happening on the road.

Findings

Larger vehicles such as trucks and buses were associated with the largest increases in drivers' heart rate and with more negative facial expressions, and shorter, more variable following distances were associated with more abrupt heart rate increases. Higher-speed highway driving was associated with more positive expressions and fewer abrupt heart rate increases, while harsh braking, acceleration and curves showed a higher share of abnormally high heart rate. State-space models estimated stress and workload from heart rate, gaze and facial movements, and suggested that traffic density and secondary tasks may shift these states, with individual drivers affected differently.

Team

Arash Tavakoli (alumnus) and Arsalan Heydarian.

Funded by

See every ORCL award on the Funding page.

Key Publications

• Tavakoli, A., Kumar, S., Guo, X., Boukhechba, M., Balali, V., & Heydarian, A. (2021). HARMONY: A human-centered multimodal driving study in the wild. IEEE Access. doi:10.1109/ACCESS.2021.3056007
• Tavakoli, A., & Heydarian, A. (2022). Multimodal driver state modeling through unsupervised learning. Accident Analysis & Prevention. doi:10.1016/j.aap.2022.106640
• Tavakoli, A., Boker, S., & Heydarian, A. (2022). Driver state modeling through latent variable state space framework in the wild. IEEE Transactions on Intelligent Transportation Systems. doi:10.1109/TITS.2022.3221858
• Tavakoli, A., Lai, N., Balali, V., & Heydarian, A. (2023). How are drivers' stress levels and emotions associated with the driving context? A naturalistic study. Journal of Transport & Health. doi:10.1016/j.jth.2023.101649

Header figure: in-cabin camera view with face and body-pose tracking, cropped from a figure in Tavakoli et al., HARMONY: A human-centered multimodal driving study in the wild, IEEE Access, 2021, CC BY 4.0.