Multi-criteria Selection of Rehearsal Samples for Continual Learning

Retaining a small subset to replay is a direct and effective way to prevent catastrophic forgetting in continual learning. However, due to data complexity and restricted memory, picking a proper subset for rehearsal is challenging and still being explored. In this work, we present a Multi-criteria Subset Selection approach that can stabilize and advance replay-based continual learning. The method picks rehearsal samples by integrating multiple criteria, including distance to prototype, intra-class cluster variation, and classifier loss. By doing so, it maximizes the comprehensive representation power of the sampled subset by ensuring its representativeness, diversity, and discriminability. We empirically find that singular criteria are likely to fail in particular tasks, while multi-criteria minimizes this risk and stabilizes task training throughout the continual learning process. Moreover, our method improves replay-based methods consistently and achieves state-of-the-art performance on both CIFAR100 and Tiny-Imagenet datasets.