Adaptive Voice Communication in Emotion-Aware Digital Companions

Abstract

This paper presents an adaptive voice communication system for emotion-aware digital companions that dynamically responds to users' affective states through expressive speech and synchronized 3D avatar animation. The system integrates real-time voice input, emotion recognition, and context-aware dialogue generation using GPT-3.5, followed by emotional text-to-speech synthesis via neural TTS. Lip-sync data is generated using phoneme alignment and rendered in sync with the avatar's facial expressions and gestures. To enhance user trust and engagement, the avatar visually mirrors the emotional tone of the speech. A cultural adaptation layer is introduced to align voice output and speech style with Sri Lankan communication norms, including tone, pacing, and formality. Implemented using a Node.js backend and React + Three.js frontend, the system demonstrates strong potential for emotionally intelligent, culturally adaptive AI interactions. This work contributes a modular pipeline for building empathetic voice agents capable of enhancing realism and trust in human-AI communication.