Collaborative Service of On-Device Unimodal Model and Cloud-Based Multimodal Model for Mobile Livestreaming Content Understanding

To guide consumers to mobile livestreams that involve their interested products, it is necessary to have a good understanding of livestreaming contents, and the key task is to accurately recognize the products being promoted by streamers. The mainstream cloud-based service framework is challenged by the high concurrency of requests, the high overhead of multimodal recognition, and the requirement of low latency. To break these bottlenecks, we propose a new device-cloud collaborative service framework, where each streamer’s mobile device holds unimodal model that can process most of video frames and also uploads the extracted unimodal features to facilitate the cloud-side multimodal recognition of the remaining few frames. On-device unimodal model is further incrementally trained over the samples constructed by leveraging the streamers’ manual labeling behaviors, leading to personalized version that can adapt to the heterogeneous and dynamic livestreaming contents of different streamers. Nevertheless, the device-side personalized unimodal features are misaligned in feature space and cannot be directly fused into the cloud-side multimodal model. We thus design a pluggable prompt generation module to transform the personalized unimodal features into prompt embeddings and prepend them to the original inputs of the multimodal backbone network, instructing feature fusion in a streamer-specific manner. We extensively evaluate on the public Fashion-Gen dataset and an industrial dataset collected from Taobao Live. We also perform online testing and practical overhead testing in Taobao Live. Evaluation results reveal the effectiveness and efficiency of our design as well as the consistent advantage over existing baselines.