A User-oriented Ensemble Method for Multi-Modal Emotion Recognition

Abstract

Emotions play a vital role in mental and physical activities of human lives. One of the biggest challenges in Human-Computer Interaction is emotion recognition. With the resurgence in the fields of Artificial Intelligence and Machine learning, a considerable number of studies have been carried out in order to address the challenge of emotion recognition. The individual heterogeneity of expressing emotions is a key problem that needs to be addressed in accurately detecting the emotional state of an individual. The purpose of this work is to propose a novel ensemble method to predict the emotions using a multimodal approach. The presented multimodal approach with the modalities of facial expressions, voice variations and, speech and social media content, are used to identify seven emotional states: anger, fear, disgust, happiness, sadness, surprise and neutral emotion. In this study, for the facial expression-based emotion recognition and voice variation-based emotion recognition, Deep Neural Network models have been used, and for emotion recognition using speech and social media content, Multinomial Naïve Bayesian algorithm is used. The mentioned three modalities were integrated using a novel ensemble method that captures the heterogeneity of individuals in how they express their emotions. The proposed ensemble method was evaluated with respect to real states of human emotions of a sample user group and the experimental results suggest that the suggested ensemble method may be more accurate in recognizing emotions. Accurate recognition of emotions may have myriad applications in domains such as healthcare, advertising and human resource management.