Facial Emotion Recognition via Label Distribution Learning and Customized Convolutional Layers

Abstract

This thesis attempts to investigate the task of recognizing human emotions from facial expressions in images, a topic that has been interest of to researchers in computer vision and machine learning. It addresses the challenge of deciphering a mixture of six basic emotions—happiness, sadness, anger, fear, surprise, and disgust—each presented with distinct intensities. This thesis introduces three Label Distribution Learning (LDL) frameworks to tackle this. Previous studies have dealt with this challenge by using LDL and focusing on optimizing a conditional probability function that attempts to reduce the relative entropy of the predicted distribution with respect to the target distribution, which leads to a lack of generality of the model. First, we propose a deep learning framework for LDL, utilizing convolutional neural network (CNN) features to broaden the model’s generalization capabilities. Named EDL-LBCNN, this framework integrates a Local Binary Convolutional (LBC) layer to refine the texture information extracted from CNNs, targeting a more precise emotion recognition. Secondly, we propose VCNN-ELDL framework, which employs an innovative Visibility Convolutional Layer (VCL). The VCL is engineered to maintain the advantages of traditional convolutional (Conv) layers for feature extraction, while also reducing the number of learnable parameters and enhancing the capture of crucial texture features from facial images. Furthermore, this research presents a novel Transformer architecture, the Visibility Convolutional Vision Transformer (VCLvT), incorporating Depth-Wise Visibility Convolutional Layers (DepthVCL) to bolster spatial feature extraction. This novel approach yields promising outcomes, particularly on limited datasets, showcasing its capacity to meet or exceed state-of-the-art performance across different dataset sizes. Through these advancements, the thesis significantly contributes to the advancement of facial emotion recognition, presenting robust, scalable models adept at interpreting the complex nuances of human emotions.