top of page


HARMONY, is a human-centered multimodal naturalistic driving study, where driver's behaviors and states are monitored through (1) in-cabin and outside video streams (2) physiological signals including driver's heart rate and hand acceleration (IMU data), (3) ambient noise, light, and the vehicle's GPS location, and (4) music logs, including song features such as tempo. HARMONY is the first study that collects long-term naturalistic facial, physiological, and environmental data simultaneously. The data is collected from 22 participants over the course of a one year study. The study is conducted by the BRAIn Lab at UVA where I am earned my Ph.D. working with Professor Heydarian.


HARMONY includes both inside and outdoor conditions through camera recordings. The indoor recordings are used for driver's emotion detection, facial expression monitoring, gaze direction, and pose direction through conventional off-the-shelf computer vision software such as OpenFace and OpenPose


The outdoor recordings are used for object detection on road for detecting matters such as number of vehicles, traffic density, type of other road users (e.g., bikes), and the distance to each object. In doing so I use off-the-shelf object detection algorithms such as MASK RCNN.  

The external contextual data is then fused with driver-specific measures such as heart rate to understand the interplay of the environment and the driver's state. For instance, a slow lead vehicle can be accompanied by an increase in heart rate.

Interesting to note that this data has been collected from 22 participants for almost one year of study. Such data can help us better visualize the effect of each infrastructural element such as intersections as well as road types and other environmental factors on drivers' well-being. 

For more information read our paper on this project at IEEE ACCESS

bottom of page