SkeletonMamba : a lightweight Mamba-based architecture for action recognition

Abstract

In the past few years, graph neural networks (GNNs) have become the preferred way to tackle the computer vision task of action recognition using skeleton data. These are usually combined with temporal operations aimed at capturing the dynamics induced by the sequence of frames. Recent works have proposed to use Mamba, a recently introduced sequence modeling architecture that offers a fast and computationally efficient alternative to Transformers, to capture these temporal dynamics, while employing GNNs to capture spatial information. Unfortunately, these approaches fail to fully untap the computational efficiency of Mamba and only achieve substandard accuracy. In this paper, we propose SkeletonMamba, a novel hybrid GNN-Mamba model combining a lightweight GNN module with a novel Temporal Mamba block. The synergy between these two components allows our model to overcome the shortcomings of existing Mamba-based action recognition architectures. We perform an extensive set of experiments demonstrating that SkeletonMamba achieves a competitive performance against SOTA approaches on the widely used NTU RGB+D 60 & 120 datasets, while minimizing the number of floating point operations and trainable parameters. Our code is publicly available at https://github.com/Spanchsan/URIS_HarmAssessment.