Multimodal Attentive Learning for Real-time Explainable Emotion Recognition in Conversations

Human emotion recognition plays a pivotal role in building an intelligent conversational agent for providing real-time automated support service in various problem settings. Recent research works have explored the temporal patterns in conversations to enable a comprehensive understanding of the content and context of conversations from a video clip, which does not fully leverage the multi-modal (facial expressions of the participants, speech tone, content, and context of the discussion) information and their temporal evolution. To address this, we propose a multimodal attentive learning framework that keeps track of spatio-temporal states of the participants and their conversation dynamics. By designing a novel contrastive loss-based optimization framework, the proposed method exhibits promise in identifying the emotion state of the individual speaker in real-time and can identify top-k words in the conversation that influence emotion recognition. The consistent superior performance over other state-of-the-art works in two large-scale datasets, MELD and IEMOCAP, demonstrate the feasibility of our approach.