Few-shot action recognition with implicit temporal alignment and pair similarity optimization

Congqi Cao, Yajuan Li, Qinyi Lv, Peng Wang, Yanning Zhang

Research output: Contribution to journalArticlepeer-review

22 Scopus citations

Abstract

Few-shot learning aims to recognize instances from novel classes with few labeled samples, which has great value in research and application. Although there has been a lot of work in this area recently, most of the existing work is based on image classification tasks. Video-based few-shot action recognition has not been explored well and remains challenging: (1) the differences of implementation details among different papers make a fair comparison difficult; (2) the wide variations and misalignment of temporal sequences make the video-level similarity comparison difficult; (3) the scarcity of labeled data makes the optimization difficult. To solve these problems, this paper presents (1) a specific setting to evaluate the performance of few-shot action recognition algorithms; (2) an implicit sequence-alignment algorithm for better video-level similarity comparison; (3) an advanced loss for few-shot learning to optimize pair similarity with limited data. Specifically, we propose a novel few-shot action recognition framework that uses long short-term memory following 3D convolutional layers for sequence modeling and alignment. Circle loss is introduced to maximize the within-class similarity and minimize the between-class similarity flexibly towards a more definite convergence target. Instead of using random or ambiguous experimental settings, we set a concrete criterion analogous to the standard image-based few-shot learning setting for few-shot action recognition evaluation. Extensive experiments on two datasets demonstrate the effectiveness of our proposed method.

Original languageEnglish
Article number103250
JournalComputer Vision and Image Understanding
Volume210
DOIs
StatePublished - Sep 2021

Keywords

  • Few-shot action recognition
  • Implicit alignment
  • Similarity optimization
  • Temporal modeling

Fingerprint

Dive into the research topics of 'Few-shot action recognition with implicit temporal alignment and pair similarity optimization'. Together they form a unique fingerprint.

Cite this