LifelongSkill: Toward Modality-Varying Lifelong Learning with Latent Knowledge Hypergraph

Human intelligence can continuously and adaptively build multimodal cognition from a series of diverse modalities in the external world. Modality-varying Continual Learning (MVCL) aims to imitate such human intelligence, which trains models on a stream of non-stationary and modality-fluctuating data distributions while sequentially transferring and protecting past knowledge. When an MVCL learner cannot anticipate the complexity of future new modalities and inter-modal interactions, the challenge of dealing with knowledge saturation (KS) with satisfactory parameter efficiency (PE) increases. Existing works focused mainly on overcoming the forgetting of past knowledge but overlooked the critical tradeoff between KS and PEC To address this gap, we propose a novel continual learning frame-work, namely LifelongSkill, that explicitly optimizes this tradeoff. Our key idea is to capture the interpretable inter-task diversity underlying the task stream, and then use this information to guide the parameter-efficient knowledge transfer and necessary network expansion. Specifically, we learn a Latent Knowledge Hypergraph (LKGraph), comprising a variety of semantically-distinct functional capabilities (namely skills) learned from tasks, to represent task diversity through skill co-occurrences. Then, we propose a Skill-wise Node Decoder (SND) to facilitate parameter-efficient network expansion and knowledge transfer guided by LKGraph. Experiment results demonstrate the proposed approach achieves the best tradeoffs between performance and parameter efficiency compared with baselines.