Automated Message Analyses Using Transformer-based Models and Their Applications

Abstract

Mental or emotional states are an important part of the information that can be observed from the speakers or writers of many discoures. On social media, people often share their emotional responses to events, stories, news, etc. The fundamental goal of this dissertation is to identify and analyze various types of mental or emotional states. The freedom of accessibility and speech that social media provides, allows people to express their feelings and beliefs, making it possible to judge users' personality and mental state. Traditional methods to identify mental health issues rely on self-reported methods in well-controlled, decontextualized environments. However, social media can be used as a new lens to explore and understand peoples' behavior more accurately in daily basis routine, where they share and exchange their feeling, opinions, plans, and even some private information such as medical diagnosis. Different transformer-based models are utilized to predict these types of mental states on social media content.

Accurately modeling mental health issues in social media has many important applications to personal health individually and society wellness collectively. In this dissertation, I propose different transformer-based models to predict these types of mental states on social media content. Building models that can capture left and right contexts jointly in sentences are necessary to enrich word representation and this is why transformer-based models like BERT and XLNet are used in this study. Utilizing multiple-head self attentions via multi-layer transformers, these models are able to model negations and other semantic relationships by paying attention to crucial words, leading to more accurate predictive models for optimism and pessimism, and suicide. It is wildly recognized and observed that deep models suffer from issues like overfitting phenomena which preventing the model from generalizing to new data. We propose a novel method called Soft Label Assignment, which has shown significant improvements to the model performance and mitigates the overfitting issue. Additional analyses were conducted to explore these mental health illnesses and how they influence our wellness. For example, demonstrate that positive emotions and sentiments in optimistic users are much more common while negative emotions and sentiments are prevalent more in pessimistic users.