Transferring External Knowledge to Weakly Supervised Temporal Action Localization

Weak supervised temporal action localization is a task with incomplete supervision, aims to localize action instances under video-level action label supervision. Despite significant progress in recent years, there are still issues of context confusion and local localization, mainly due to the inconsistent goals between the classification model and the localization task. Essentially, it is a problem where coarse-grained label information is difficult to ensure alignment between instance-level data and video-level labels. This problem mainly stems from the lack of precise annotation information, which limits the performance of the task. To address this issue, we propose the incorporation of two types of external knowledge: explicit knowledge, such as a knowledge graph, which aids in extracting intricate action details from label semantics, and tacit knowledge acquired from pre-trained models, facilitating the optimal alignment of vision and text through the utilization of rich potential information. In this article, we initially introduced two types of knowledge into the benchmark method framework separately. Subsequently, we aimed to integrate this knowledge effectively to introduce additional information. Our approach is implemented using a modular, plug-and-play design, which allows for the seamless integration of knowledge into various methods, rendering it an efficient and flexible endeavor. In addition, the experimental results indicate that our method improves performance on the THUMOS’14 dataset and two baseline models.