top of page


Driver State Modeling through Latent Variable State Space Framework in the Wild

Abstract: Analyzing the impact of the environment on drivers’ stress level and workload is of high importance for designing human-centered driver-vehicle interaction systems and to ultimately help build a safer driving experience. However, driver’s state, including stress level and workload, are latent variables that cannot be measured on their own and should be estimated through sensor measurements such as psychophysiological measures. We propose using a latent-variable state-space modeling framework for driver state analysis. By using latent-variable state-space models, we model drivers’ workload and stress levels as latent variables estimated through multimodal human sensing data, under the perturbations of the environment in a state-space format and in a holistic manner. Through using a case study of multimodal driving data collected from 11 participants, we first estimate the latent stress level and workload of drivers from their heart rate, gaze measures, and intensity of facial action units. We then show that external contextual elements such as the number of vehicles as a proxy for traffic density and secondary task demands may be associated with changes in driver’s stress levels and workload. We also show that different drivers may be impacted differently by the aforementioned perturbations. We found out that drivers’ latent states at previous timesteps are highly associated with their current states. Additionally, we discuss the utility of state-space models in analyzing the possible lag between the two latent variables of stress level and workload, which might be indicative of information transmission between the different parts of the driver’s psychophysiology in the wild.


Collaborators: Steven Boker, and Arsalan Heydarian (Ph.D. Advisor). 

Link to the paper


Multimodal Driver State Modeling through Unsupervised Learning

Abstract: Naturalistic driving data (NDD) can help understand drivers' reactions to each driving scenario and provide personalized context to driving behavior. However, NDD requires a high amount of manual labor to label certain driver's state and behavioral patterns. Unsupervised analysis of NDD can be used to automatically detect different patterns from the driver and vehicle data. In this paper, we propose a methodology to understand changes in driver's physiological responses within different driving patterns. Our methodology first decomposes a driving scenario by using a Bayesian Change Point detection model. We then apply the Latent Dirichlet Allocation method on both driver state and behavior data to detect patterns. We present two case studies in which vehicles were equipped to collect exterior, interior, and driver behavioral data. Four patterns of driving behaviors (i.e., harsh brake, normal brake, curved driving, and highway driving), as well as two patterns of driver's heart rate (HR) (i.e., normal vs. abnormal high HR), and gaze entropy (i.e., low versus high), were detected in these two case studies. The findings of these case studies indicated that among our participants, the drivers' HR had a higher fraction of abnormal patterns during harsh brakes, accelerating and curved driving. Additionally, free-flow driving with close to zero accelerations on the highway was accompanied by more fraction of normal HR as well as a lower gaze entropy pattern. With the proposed methodology we can better understand variations in driver's psychophysiological states within different driving scenarios. The findings of this work, has the potential to guide future autonomous vehicles to take actions that are fit to each specific driver.

Collaborators:  Arsalan Heydarian

Link to the paper


Occupant Privacy Perception, Awareness, and Preferences in Smart Office Environments

Abstract: Building management systems tout numerous benefits, such as energy efficiency and occupant comfort but rely on vast amounts of data from various sensors. Advancements in machine learning algorithms make it possible to extract personal information about occupants and their activities beyond the intended design of a non-intrusive sensor. However, occupants are not informed of data collection and possess different privacy preferences and thresholds for privacy loss. While privacy perceptions and preferences are most understood in smart homes, limited studies have evaluated these factors in smart office buildings, where there are more users and different privacy risks. To better understand occupants' perceptions and privacy preferences, we conducted twenty-four semi-structured interviews between April 2022 and May 2022 on occupants of a smart office building. We found that data modality features and personal features contribute to people's privacy preferences. The features of the collected modality define data modality features -- spatial, security, and temporal context. In contrast, personal features consist of one's awareness of data modality features and data inferences, definitions of privacy and security, and the available rewards and utility. Our proposed model of people's privacy preferences in smart office buildings helps design more effective measures to improve people's privacy.


Collaborators: Beatrice Li, and Arsalan Heydarian 

Link to the paper


Human wellbeing responses to real and simulated workplaces: A comparison of in-person, online, and virtual environments

Abstract: Recent studies have emphasized the role of building design on occupant wellbeing. However, studying the impact of built features on wellbeing is time-consuming and expensive. Our work explores the value of different methods to simulate workplace environments and their impact on wellbeing outcomes. Following a laboratory experiment that highlighted the potential of windows and natural materials to reduce stress, we conducted an immersive online replication to assess the continuity of results on a different platform. Online participants reported lower negative affect with natural materials compared to artificial materials, and higher positive affect in the presence of windows vs no window condition, making the stress results similar to those in the lab. Additionally, windows and diverse representations promoted belonging and creativity, respectively. A virtual reality (VR) replication is currently underway with identical variables to investigate the role of VR in facilitating research in this field. Our work contributes to a better understanding of the value of different workplaces (e.g., office, hybrid, or remote) based on their design characteristics.


Collaborators: Eva Bianchi, Basma Altaf, Isabella Douglas, James Landay, and Sarah Billington

Link to the paper


Rethinking infrastructure design: Evaluating pedestrians and VRUs' psychophysiological and behavioral responses to different roadway designs

The integration of human-centric approaches has gained more attention recently due to more automated systems being introduced into our built environments (buildings, roads, vehicles, etc.), which requires a correct understanding of how humans perceive such systems and respond to them. This paper introduces an Immersive Virtual Environment-based method to evaluate the infrastructure design with psycho-physiological and behavioral responses from the vulnerable road users, especially for pedestrians. A case study of pedestrian mid-block crossings with three crossing infrastructure designs (painted crosswalk, crosswalk with flashing beacons, and a smartphone app for connected vehicles) are tested. Results from 51 participants indicate there are differences between the subjective and objective measurement. A higher subjective safety rating is reported for the flashing beacon design, while the psychophysiological and behavioral data indicate that the flashing beacon and smartphone app are similar in terms of crossing behaviors, eye tracking measurements, and heart rate. In addition, the smartphone app scenario appears to have a lower stress level as indicated by eye tracking data, although many participants don't have prior experience with it. Suggestions are made for the implementation of new technologies, which can increase public acceptance of new technologies and pedestrian safety in the future.

Collaborators: Xiang Guo, Austin Angulo, Erin Robartes, Donna Chen, and Arsalan Heydarian (Ph.D. Advisor). 

Read the article here

More information about this project: see this webpage

HARMONY: A Human-centered Multimodal Driving Study in the Wild

Abstract: Effective shared autonomy requires a clear understanding of driver's behavior, which is governed by multiple psychophysiological and environmental variables. Disentangling this intricate web of interactions requires understanding the driver's state and behaviors in different real-world scenarios, longitudinally. Naturalistic Driving Studies (NDS) have shown to be an effective approach to understanding the driver's state and behavior in real-world scenarios. However, due to the lack of technological and computing capabilities, former NDS only focused on vision-based approaches, ignoring important psychophysiological factors such as cognition and emotion. The main objective of this paper is to introduce HARMONY, a human-centered multimodal naturalistic driving study, where driver's behaviors and states are monitored through (1) in-cabin and outside video streams (2) physiological signals including driver's heart rate and hand acceleration (IMU data), (3) ambient noise, light, and the vehicle's GPS location, and (4) music logs, including song features such as tempo. HARMONY is the first study that collects long-term naturalistic facial, physiological, and environmental data simultaneously. This paper summarizes HARMONY's goals, framework design, data collection and analysis, and the on-going and future research efforts. Through a presented case study, we first demonstrate the importance of longitudinal driver state sensing through using Kernel Density Estimation Methods. Second, we leverage the application of Bayesian Change Point detection methods to demonstrate how we can identify driver behaviors and responses to the environmental conditions by fusing psychophysiological information with features extracted from video streams.


Collaborators: Shashwat Kumar, Xiang Guo, Vahid Balali, Mehdi Boukhechba, and Arsalan Heydarian (Ph.D. Advisor). 

Link to the paper

Link to the video presentation

Link to the project page


The Impact of Surrounding Road Objects and Conditions on Drivers Abrupt Heart Rate Changes

Abstract: Recent studies have pointed out the importance of mitigating drivers stress and negative emotions. These studies show that certain road objects such as big vehicles might be associated with higher stress levels based on drivers subjective stress measures. Additionally, research shows strong correlations between drivers stress levels and increased heart rate (HR). In this paper, based on a naturalistic multimodal driving dataset, we analyze the visual scenes of driving in the vicinity of abrupt increases in drivers HR for the presence of certain stress-inducing road objects. We show that the probability of the presence of such objects increases when becoming closer to the abrupt increase in drivers HR. Additionally, we show that drivers facial engagement changes significantly in the vicinity of abrupt increases in HR. Our results lay the ground for a human-centered driving experience by detecting and mitigating drivers stress levels in the wild.

Collaborators:  Arsalan Heydarian. 

Link to the paper


How are Drivers' Stress Levels and Emotions Associated with the Driving Context? A Naturalistic Study

Abstract: Understanding and mitigating drivers' negative emotions, and stress levels, is of high importance for enhancing road safety, and human well-being. While detecting drivers' stress and negative emotions can significantly help with this goal, understanding what might be associated with increases in drivers' negative emotions and high stress level, might better help with planning interventions, which has not been explored in detail. Methods: In this study, by using a naturalistic driving study database, we analyze the changes in drivers' heart rate and facial expressions with respect to the changes in the driving scene, including road objects and the dynamical relationship between the ego vehicle and the lead vehicle. Results: Our results indicate that different road objects might be associated with varying levels of increase in drivers' HR as well as different proportions of negative facial emotions detected through computer vision. Larger vehicles on the road, such as trucks and buses, are associated with the highest amount of increase in drivers' HR as well as negative emotions. Additionally, shorter distances and higher standard deviation in the distance to the lead vehicle are associated with a higher number of abrupt increases in drivers’ HR, depicting a possible increase in stress level. Our finding indicates more positive emotions, lower facial engagement, and a lower abrupt increase in HR in highway environments. Conclusion: This research collectively shows that certain road environments such as highways with less stress-inducing objects are much more suited to promote drivers well-being and should be considered in routing use cases.

Collaborators:  Nathan Lai, Vahid Balali, and Arsalan Heydarian. 

Link to the paper


Psycho-physiological measures on a bicycle simulator in immersive virtual environments: how protected/curbside bike lanes may improve perceived safety

Abstract: As a healthier and more sustainable way of mobility, cycling has been advocated by literature and policy. However, current trends in bicyclist crash fatalities suggest deficiencies in current roadway design in protecting these vulnerable road users. The lack of cycling data is a common challenge for studying bicyclists’ safety, behavior, and comfort levels under different design contexts. To understand bicyclists’ behavioral and physiological responses in an efficient and safe way, this study uses a bicycle simulator within an immersive virtual environment (IVE). Off-the-shelf sensors are utilized to evaluate bicyclists’ cycling performance (speed and lane position) and physiological responses (eye tracking and heart rate). Participants bike in a simulated virtual environment modeled to scale from a real-world street with a shared bike lane (sharrows) to evaluate how the introduction of a curbside bike lane and a protected bike lane with flexible delineators may impact perceptions of safety, as well as behavioral and psycho-physiological responses. Results from 50 participants (representing both genders and across a wide age range) show that the protected bike lane design received the highest perceived safety rating and exhibited the lowest average cycling speed. Furthermore, both the curbside bike lane and the protected bike lane scenarios show a less dispersed gaze distribution than the as-built sharrows scenario, reflecting a higher gaze focus among bicyclists on the biking task in the curbside bike lane and protected bike lane scenarios, compared to when bicyclists share right of way with vehicles. Additionally, heart rate change point results from the study suggest that creating dedicated zones for bicyclists (curbside bike lanes or protected bike lanes) has the potential to reduce bicyclists’ stress levels. Female participants show a higher preference on the protected bike lane design and a lower perceived safety rating on the sharrows. These findings are from participants riding a bicycle simulator in an immersive virtual environment and inform, but do not fully reflect, bicycling in the real world.

Collaborators: Xiang Guo, Erin Robartes, Austin Angulo, Donna Chen, and Arsalan Heydarian (Ph.D. Advisor). 

Full paper can be accessed here

More information about this project: see this webpage

System_framework_2_notext (1).jpg

Leveraging Ubiquitous Computing for Empathetic Routing: A Naturalistic Data-driven Approach

Abstract: Although we extensively use routing services in our daily commutes, such systems are yet to be personalized around the user. It often happens that different routes are close in their estimated time of arrival (ETA) while being very different in how they affect the driver’s states. Using traces of a user’s physiological measures, different candidate routes can be ranked based on how they affect users’ well-being. In this research, we introduce the “empathetic routing” framework for providing human-centered routing based on historical biomarkers of the drivers collected through naturalistic settings and by using smart wearable devices. Through this framework, we rank three specific routes between two points in the city of Charlottesville, based on historical driver heart rate data collected through a three-month naturalistic driving study. Additionally, we demonstrate that the proposed framework is capable of finding infrastructural elements in a route that can potentially affect a driver’s well-being.

Collaborators:  Mehdi Boukhechba, and Arsalan Heydarian. 

Link to the paper

Link to the video presentation

Link to the project page


Driver State and Behavior Detection Through Smart Wearables

Abstract: Integrating driver, in-cabin, and outside environment's contextual cues into the vehicle's decision making is the centerpiece of semi-automated vehicle safety. Multiple systems have been developed for providing context to the vehicle, which often rely on video streams capturing drivers' physical and environmental states. While video streams are a rich source of information, their ability in providing context can be challenging in certain situations, such as low illuminance environments (e.g., night driving), and they are highly privacy-intrusive. In this study, we leverage passive sensing through smartwatches for classifying elements of driving context. Specifically, through using the data collected from 15 participants in a naturalistic driving study, and by using multiple machine learning algorithms such as random forest, we classify driver's activities (e.g., using phone and eating), outside events (e.g., passing intersection and changing lane), and outside road attributes (e.g., driving in a city versus a highway) with an average F1 score of 94.55, 98.27, and 97.86 % respectively, through 10-fold cross-validation. Our results show the applicability of multimodal data retrieved through smart wearable devices in providing context in real-world driving scenarios and pave the way for a better shared autonomy and privacy-aware driving data-collection, analysis, and feedback for future autonomous vehicles.

Collaborators: Shashwat Kumar, Mehdi Boukhechba, and Arsalan Heydarian. 

Link to the paper

Link to the video presentation


Personalized Driver State Profiles: A Naturalistic Data-Driven Study

Abstract: Previous studies suggest that variation in driver’s states, such as being under stress, can degrade drivers’ performance. Moreover, different drivers may have varying behaviors and reactions in different road conditions and environments (contexts). Thus, personalized driver models given different contextual settings can assist in better predicting the drivers’ states (behavioral and psychological); this can then allow vehicles to adjust the driving experience around the driver and passengers’ preferences and comfort levels. This paper aims at developing personalized hierarchical driver’s state models by considering driver’s heart rate variability (HRV) in relation to the changes in various contextual settings of road, weather, and presence of a passenger. Results from 12 participants over 150 h of driving data suggest that drivers are on average less stressed in highways compared to cities, when being with a passenger compared to alone, and when driving in non-rainy conditions compared to rainy weather.

Collaborators: Mehdi Boukhechba, and Arsalan Heydarian. 

Link to the paper


How do Environmental Factors Affect Drivers’ Gaze and Head Movements?

Abstract: Studies have shown that environmental factors affect driving behaviors. For instance, weather conditions and the presence of a passenger have been shown to significantly affect the speed of the driver. As one of the important measures of driving behavior is the gaze and head movements of the driver, such metrics can be potentially used towards understanding the effects of environmental factors on the driver’s behavior in real-time. In this study, using a naturalistic study platform, videos have been collected from six participants for more than four weeks of a fully naturalistic driving scenario. The videos of both the participants’ faces and roads have been cleaned and manually categorized depending on weather, road type, and passenger conditions. Facial videos have been analyzed using OpenFace to retrieve the gaze direction and head movements of the driver. Results, overall, suggest that the gaze direction and head movements of the driver are affected by a combination of environmental factors and individual differences. Specifically, results depict the distracting effect of the passenger on some individuals. In addition, it shows that highways and city streets are the cause for maximum distraction on the driver’s gaze. 

Collaborators: Vahid Balali, and Arsalan Heydarian. 

Link to the paper


Flood mitigation in coastal urban catchments using real-time stormwater infrastructure control and reinforcement learning

Abstract: Flooding in coastal cities is increasing due to climate change and sea-level rise, stressing the traditional stormwater systems these communities rely on. Automated real-time control (RTC) of these systems can improve performance, and creating control policies for smart stormwater systems is an active area of study. This research explores reinforcement learning (RL) to create control policies to mitigate flood risk. RL is trained using a model of hypothetical urban catchments with a tidal boundary and two retention ponds with controllable valves. RL's performance is compared to the passive system, a model predictive control (MPC) strategy, and a rule-based control strategy (RBC). RL learns to proactively manage pond levels using current and forecast conditions and reduced flooding by 32% over the passive system. Compared to the MPC approach using a physics-based model and genetic algorithm, RL achieved nearly the same flood reduction, just 3% less than MPC, with a significant 88× speedup in runtime. Compared to RBC, RL was able to quickly learn similar control strategies and reduced flooding by an additional 19%. This research demonstrates that RL can effectively control a simple system and offers a computationally efficient method that could scale to RTC of more complex stormwater systems.

Collaborators: Benjamine Bowes, Cheng Wang, Arsalan Heydarian, Madhur Behl, Peter Beling, and Jonathan Goodall


Link to the paper


Abstract: Studies have indicated that emotions can significantly be influenced by environmental factors; these factors can also significantly influence drivers’ emotional state and, accordingly, their driving behavior. Furthermore, as the demand for autonomous vehicles is expected to significantly increase within the next decade, a proper understanding of drivers’/passengers’ emotions, behavior, and preferences will be needed in order to create an acceptable level of trust with humans. This paper proposes a novel semi-automated approach for understanding the effect of environmental factors on drivers’ emotions and behavioral changes through a naturalistic driving study. This setup includes a frontal road and facial camera, a smart watch for tracking physiological measurements, and a Controller Area Network (CAN) serial data logger. The results suggest that the driver’s affect is highly influenced by the type of road and the weather conditions, which have the potential to change driving behaviors. For instance, when the research defines emotional metrics as valence and engagement, results reveal there exist significant differences between human emotion in different weather conditions and road types. Participants’ engagement was higher in rainy and clear weather compared to cloudy weather. Moreover, engagement was higher on city streets and highways compared to one-lane roads and two-lane highways.

Collaborators: Vahid Balali, and Arsalan Heydarian. 

Link to the paper



Abstract: The increasing urbanization of coastal regions makes beach erosion and coastline protection an important field of research (Elko et al., 2014). Excess pore pressures and pore pressure gradients in the soil matrix can impact sediment mobilization and erosion in terms of liquefaction (Sumer, 2014). Despite previous studies, there are still unsolved questions regarding coastal liquefaction due to wave action. Particularly, the role of groundwater dynamics, the impact of wave breaking, sediment reorganization, and potential air content represent unsolved problems. Furthermore, open questions still exist regarding the interaction and roles of excess pore pressure built-up, vertical pressure gradients and horizontal pressure gradients (Foster et al., 2006; Yeh and Mason, 2014; Sumer, 2014; Stark, 2017). We hypothesize that temperature variations may reveal complementary information with regard to pore water fluid behavior, such as pore space saturation, groundwater flows, exfiltration and infiltration processes, and impact of wave forcing. The study presented here shows some preliminary data sets of combined pore pressure and temperature recordings.

Collaborators: Nina Stark (MSc. Advisor), Alex E. Hay


Link to the paper

bottom of page