Online action recognition using covariance of shape and motion

Igor Kviatkovsky, Ehud Rivlin, Ilan Shimshoni

Research output: Contribution to journalArticlepeer-review


We propose a novel approach for online action recognition. The action is represented in a low dimensional (15D) space using a covariance descriptor of shape and motion features - spatio-temporal coordinates and optical flow of pixels belonging to extracted silhouettes. We analyze the applicability of the descriptor for online scenarios where action classification is performed based on incomplete spatio-temporal volumes. In order to enable our online action classification algorithm to be applied in real time, we introduce two modifications, namely the incremental covariance update and the on demand nearest neighbor classification. In our experiments we use quality measures, such as latency, especially designed for the online scenario to report the algorithm's performance. We evaluate the performance of our descriptor on standard, publicly available datasets for gesture recognition, namely the Cambridge-Gestures dataset and the ChaLearn One-Shot-Learning dataset and show that its performance is comparable to the state-of-the-art despite its relative simplicity. The evaluation on the UCF-101 action recognition dataset demonstrates that the descriptor is applicable in challenging unconstrained environments.

Original languageEnglish
Pages (from-to)15-26
Number of pages12
JournalComputer Vision and Image Understanding
StatePublished - 2014

Bibliographical note

Publisher Copyright:
© 2014 Elsevier Inc. All rights reserved.


  • Covariance descriptor
  • Gesture recognition
  • Incremental covariance computation
  • Online action recognition

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • Computer Vision and Pattern Recognition


Dive into the research topics of 'Online action recognition using covariance of shape and motion'. Together they form a unique fingerprint.

Cite this