Sentiment-Driven Health Messaging: Comparing DistilBERT and VADER on Timed Narratives with Animated Educational Videos using Manim

Abstract

Abstract— This project builds two educational animations using Manim and explores whether general-purpose sentiment analysis tools can help estimate the “healthiness” of child-friendly, short health education videos, by defining a script with timestamped segments labelled as Healthy or Unhealthy. The focus is on comparing a transformer-based sentiment classifier (DistilBERT SST-2) with a rule-based tool (VADER). DistilBERT links POSITIVE sentiment to Healthy and NEGATIVE to Unhealthy. VADER uses a score threshold (≥0.05 for Healthy, ≤−0.05 for Unhealthy, and values in between as Neutral). For evaluation, the “Neutral” predictions are converted to “Unhealthy”. Precision, recall, and F1 scores (per video and overall) are calculated, and confusion matrices are used to visualize the performance. In the custom dataset of 24 segments, both methods perform well when the language is clearly positive or negative. DistilBERT rarely produces Neutral labels but sometimes misinterprets healthy advice that includes negation. VADER, However, often predicts Neutral, which is penalized in the evaluation method. The study also discusses the limitations of using sentiment as a proxy for healthfulness, including the effects of class imbalance and how Neutral predictions are handled. Few potential improvements are suggested, such as domain specific model tuning, more accurate threshold settings, and treating Neutral as an undecided, or “abstain” option.