Multimodal Conversatioal Emotion Analysis with Robustness to Incomplete Modality Details

Emotion recognition's efficacy for individual and group conversations hinges upon integrating data from diverse modalities (e.g., text, audio, and video). However, the intrinsic heterogeneity of these data sources, coupled with the interpersonal contextual influences in groups, present significant challenges to accurately identifying emotion. Furthermore, during inference most applications cannot guarantee the presence of all modalities the model was trained on. Towards this, we propose Missing Information robust Joint Transformer (MIJoT), which is designed to capture the extensive cross-modal spatio-temporal context in a dialogue. Unlike a majority of existing works, which rely on a fundamental assumption that a full set of modalities is present during inference, MIJoT's Masked Mode-specific Data Modelling ensures consistent performance in presence of partial modality details. By enabling powerful masked modeling pretraining, MIJoT delivers an impressive ability to deal with a wide range of incomplete modality scenarios. We also utilize a Contrastive Imbalance Loss that maximizes inter-class distance on a high-dimensional hyper-sphere, to improve performance on minority classes. An effective multimodal fusion enables the model to analyze both utterances and dialogue (sequence of utterances) between multiple people within an integrated computation framework and deliver superior performance (4-12%) improvement in weighted-F1 scores) compared to the current best-performing methods across several large-scale public datasets. Additionally, MIJoT with incomplete modalities report comparable performance to existing state-of-the-art techniques which use complete multi-mode information, and thereby demonstrate its robustness to partial mode information of the queries.