Interactive Sign Language Learning System with Multimodal Emotion and Interaction Recognition Analysis

Abstract

Child-centric intelligent applications increasingly require adaptive mechanisms to understand emotional engagement and provide inclusive learning experiences. However, existing systems often rely on single-modal emotion recognition or non-interactive learning approaches, limiting reliability and accessibility. This paper presents a twocomponent intelligent framework that addresses these challenges through a multimodal emotion and interaction analysis module that fuses facial emotion recognition and hand movement intensity, and a gamified sign language learning and progress-tracking module designed for deaf and mute children. The first component employs an EfficientNet-B0-based facial emotion recognition model trained on FER2013 and fine-tuned using CK+, achieving 96.9% validation accuracy, combined with motion-based hand interaction analysis to generate stable session-level engagement insights. The second component introduces a camera-assisted, game-based sign language learning environment with a landmark-based DNN gesture validation model achieving 97% validation accuracy, along with a parent-oriented progress dashboard. Experimental results demonstrate improved robustness in engagement detection and effective learning support through gamification. The proposed system highlights the novelty of integrating multimodal behavioral analysis with accessible, child-friendly learning mechanisms suitable for real-time environments.