Multimodal, Context-Aware, Feature Representation Learning for Classification and Localization

Automatic detection of malicious or provocative social media content is increasingly important to efforts leading to counteract illicit activities in the digital platform. This paper proposes a context-aware, regional feature representation learning framework that exploits multimodal cues to improve automatic detection by enhancing classification via accurate localization. Unlike most existing multimodal approaches, which evaluate category membership of multimedia web contents only at the image level, our approach leverages object proposals to identify potential interest regions within images in order to provide more precise localization performance that in turn, in a context-aware setting, also improves classification result. The proposed attention learning module estimates the domain specific cross-modal, fine-grained regional feature correspondence, conditioned on the classification categories, by evaluating mode relevance scores in a data-driven manner. The initial classification decision is further validated using a query adaptive decision fine-tuning for a more accurate final prediction. Experiments on publicly available datasets and on in-house datasets demonstrate superior classification performance as compared to monomodal and existing multimodal baselines.