Few-Shot Class-Incremental Learning with Meta-Learned Class Structures

Learning continually from few-shot examples is a hallmark of human intelligence but it poses a great challenge for deep neural networks since they commonly suffer from catastrophic forgetting and overfitting. In this paper, we tackle this challenge in the few-shot class-incremental learning (FSCIL) setting, where a sequence of few-shot learning sessions containing disjoint sets of classes is created for a model to incrementally learn new classes, and the model should avoid forgetting information of old classes. Simply accumulating information of all learned classes will severely degrade the performance of the model since new classes are not learned to be discriminative across different sessions and they tend to be biased with limited training examples. To address this problem, we introduce class structures to regularize the learned classes on how they should distribute in the embedding space such that they are distinctive with each other within and across different learning sessions. Concretely, these class structures are encoded in a subspace where an alignment kernel aligns a learned class with class structures by moving it along the base vectors of the subspace. We sample incremental tasks in the training to simulate incremental learning and formulate the training as a meta-learning process to learn generalizable class structures across many incremental tasks. Experimental results on the CIFAR100, miniImageNet, and CUB200 datasets demonstrate the effectiveness of our method in combating overfitting and catastrophic forgetting.