AI-Driven 3D Avatar Framework for Sign Language Translation and Gesture Representation

Abstract

The communication gap between Deaf and Hard-of-Hearing (DHH) communities and mainstream multimedia platforms remains a critical accessibility challenge. Conventional solutions such as subtitles fail to capture the expressiveness, gestures, and finger movements intrinsic to sign language. Existing AI-based avatar systems further suffer from incomplete hand articulation, unnatural body rotations, and desynchronized playback, limiting their realism and effectiveness. To address these challenges, this research introduces AuralFlix, an AI-driven framework that translates spoken or textual content into naturalistic 3D avatar animations for sign language representation. The system integrates MMPose-based pose estimation to capture full-body and hand keypoints, which are mapped to a Mixamo-compatible skeleton in Blender using quaternion-based animation. A custom animation correction pipeline minimizes rotational artifacts, improves finger articulation, and synchronizes the generated motion with video playback through a dedicated multimedia player. Experimental evaluation demonstrates an average positional accuracy of 3.5 mm and rotational error below 3°, achieving approximately 20% improvement in gesture fidelity compared to baseline markerless animation systems. The proposed framework provides a scalable, low-cost, and expressive accessibility solution that enhances digital communication for the DHH community.