HARMONY, is a human-centered multimodal naturalistic driving study, where driver's behaviors and states are monitored through (1) in-cabin and outside video streams (2) physiological signals including driver's heart rate and hand acceleration (IMU data), (3) ambient noise, light, and the vehicle's GPS location, and (4) music logs, including song features such as tempo. HARMONY is the first study that collects long-term naturalistic facial, physiological, and environmental data simultaneously. The data is collected from 22 participants over the course of a one year study. The study is conducted by the BRAIn Lab at UVA where I am earned my Ph.D. working with Professor Heydarian.
The outdoor recordings are used for object detection on road for detecting matters such as number of vehicles, traffic density, type of other road users (e.g., bikes), and the distance to each object. In doing so I use off-the-shelf object detection algorithms such as MASK RCNN.
The external contextual data is then fused with driver-specific measures such as heart rate to understand the interplay of the environment and the driver's state. For instance, a slow lead vehicle can be accompanied by an increase in heart rate.
Interesting to note that this data has been collected from 22 participants for almost one year of study. Such data can help us better visualize the effect of each infrastructural element such as intersections as well as road types and other environmental factors on drivers' well-being.