A Facial Expression-Aware Edge AI System For Driver Safety Monitoring

Abstract

This dissertation presents a driver monitoring system (DMS) that integrates emotion recognition to address critical issues in road safety. Road safety has become a global concern due to the significant increase in vehicle numbers and the rapid growth of transportation infrastructure. The number one cause of road accidents is human error, with a 90% ratio, with common contributing factors like distraction, drowsiness, panic, and fatigue. Traditional DMS approaches often fall short in identifying these emotional and cognitive states, limiting their effectiveness in accident prevention. To address these limitations, this research proposes a robust, deep-learning-based DMS framework designed to identify and respond to driver emotions and behaviors that may compromise safety. The proposed system utilizes advanced convolutional neural networks (CNN), specifically the inception module and Caffe-based ResNet-10 with a single-shot detector (SSD), to perform efficient facial detection and classification. These chosen model structures helped balance computational efficiency and accuracy. The DMS is trained on an extensive, diverse dataset comprising approximately 198,000 images and 1,600 videos sourced from multiple public and private datasets, ensuring the system’s robustness across a range of emotions and real-world driving scenarios. Emotions of interest include high-risk states such as drowsiness, distraction, and fear, alongside neutral conditions, and the model can perform well in different conditions, including low-light and foggy/blurry environments. Methodologically, the system incorporates essential data preprocessing techniques such as resizing, brightness normalization, pixel scaling, and noise reduction to optimize the model’s performance. On top of that, data augmentation and grayscale conversion improves the dataset’s variability, allowing the decrease of computational costs without sacrificing accuracy. This approach enabled the model to achieve high performance metrics, with an overall accuracy of 98.6% , an F1-score of 0.979, precision of 0.980, and recall of 0.979 across the four primary emotional states. This research contributes to the field by offering a less invasive, real-time solution for monitoring high-risk driver behaviors and providing insights for further advancements in automated driver assistance technologies. Future directions include optimizing the system for microcontrollers with low power consumption and implementing alerts for high-risk states to further mitigate accident risks, as well as including a multi-modal fusion of data from different sources (Infrared Camera, and a Microphone) to increase emotion recognition accuracy, which leads to taking better control and initiating more efficient proactive interventions.