Navigating Arabic Sentiments: An Evaluation of Multilingual and Arabic Dedicated Large Language Models

Abstract

Expressing emotions in written text, especially in Arabic with its complex structure and poetic elements, can be challenging.While body language enriches spoken communication with emotional depth, written Arabic often lacks this nuance. The advent of Large Language Models (LLMs) has revolutionized natural language processing (NLP), excelling in tasks like text generation and sentiment analysis. However, the performance of these models varies significantly depending on the language and task. Arabic poses unique challenges due to its complex morphology and diverse dialects. This research investigates the impact of LLMs, particularly those tailored for Arabic, on the emotional depth of the written text. By evaluating how these models modify expressions, the study aims to understand whether LLMs preserve or constrain the intricate emotional nuances inherent in Arabic. The findings will contribute to the development of more effective AI tools for digital communication in the Arabic-speaking world, enhancing applications in fields such as sentiment analysis, opinion mining, and content moderation. Through a comprehensive analysis of over 81,000 Arabic texts, including tweets and book reviews, the study examines the performance of the general-purpose LLM ChatGPT and the Arabic-specific LLM JAIS, focusing on the sentiment shifts introduced by their edits. The results reveal a significant tendency of these models to introduce a positive bias, reducing the frequency of extremely negative sentiments. These insights highlight the necessity of incorporating cultural and linguistic nuances into LLM training data, emphasizing the importance of responsible development and ethical considerations in LLM applications.