Abstract

Existing multimodal-based human action recognition approaches arecomputationally intensive, limiting their deployment in real-time applications.In this work, we present a novel and efficient pose-driven attention-guidedmultimodal network (EPAM-Net) for action recognition in videos. Specifically,we propose eXpand temporal Shift (X-ShiftNet) convolutional architectures forRGB and pose streams to capture spatio-temporal features from RGB videos andtheir skeleton sequences. The X-ShiftNet tackles the high computational cost ofthe 3D CNNs by integrating the Temporal Shift Module (TSM) into an efficient 2DCNN, enabling efficient spatiotemporal learning. Then skeleton features areutilized to guide the visual network stream, focusing on keyframes and theirsalient spatial regions using the proposed spatial-temporal attention block.Finally, the predictions of the two streams are fused for final classification.The experimental results show that our method, with a significant reduction infloating-point operations (FLOPs), outperforms and competes with thestate-of-the-art methods on NTU RGB-D 60, NTU RGB-D 120, PKU-MMD, and ToyotaSmartHome datasets. The proposed EPAM-Net provides up to a 72.8x reduction inFLOPs and up to a 48.6x reduction in the number of network parameters. The codewill be available athttps://github.com/ahmed-nady/Multimodal-Action-Recognition.