Learning task-shared and specific knowledge via mixture-of-experts in generative model for continual learning

Class-Incremental Learning (CIL) enables models to learn new classes from sequentially acquired data while retaining knowledge of previously learned ones. Traditional CIL methods typically rely on discriminative models, which often suffer from classifier bias when extending the classifier to accommodate new tasks. In this work, we take a different approach by leveraging recent generative multi-modal models for CIL. Unlike discriminative models, generative multi-modal models exploit strong semantic relationships between text and images for classification. This eliminates the need to expand the classifier for new classes, effectively avoiding the classifier bias issue. To further improve knowledge adaptation in this setting, we propose a Mixture-of-Experts (MoE)-based framework for rehearsal-free CIL. Our method introduces both task-specific and task-shared experts to capture knowledge unique to individual tasks as well as shared knowledge across tasks. A routing network is also designed to dynamically select the most appropriate experts for each input instance. Extensive experiments on ImageNet-R, TinyImageNet, CIFAR-100 and ImageNet100 demonstrate that our approach significantly outperforms existing state-of-the-art methods in CIL. Code will be available at https://github.com/MoE_GMCL.