Abstract

Current methods for analyzing student engagement in e-learning platforms,including automated systems, often struggle with challenges such as handlingfuzzy sentiment in text comments and relying on limited metadata. Traditionalapproaches, such as surveys and questionnaires, also face issues like smallsample sizes and scalability. In this paper, we introduce LLM-SEM (LanguageModel-Based Student Engagement Metric), a novel approach that leverages videometadata and sentiment analysis of student comments to measure engagement. Byutilizing recent Large Language Models (LLMs), we generate high-qualitysentiment predictions to mitigate text fuzziness and normalize key featuressuch as views and likes. Our holistic method combines comprehensive metadatawith sentiment polarity scores to gauge engagement at both the course andlesson levels. Extensive experiments were conducted to evaluate various LLMmodels, demonstrating the effectiveness of LLM-SEM in providing a scalable andaccurate measure of student engagement. We fine-tuned TXLM-RoBERTa usinghuman-annotated sentiment datasets to enhance prediction accuracy and utilizedLLama 3B, and Gemma 9B from Ollama.