Shuqiang Cao, Weixin Luo, Ran Song, Bairui Wang, Lin Ma, Wei Zhang
Online action detection aims to recognize actions from streaming videos in real time. Existing approaches typically rely on heavy and frozen video backbones to extract segment features, focusing primarily on modeling temporal relationships across segments. Although these methods have achieved initial success, such frameworks fail to fully leverage the benefits of trainable backbones due to the prohibitive training cost. In response, we propose E2E-LOAD, an end-to-end trainable network that employs a lightweight spatial module to encode video segments into features stored in a stream buffer for efficient reuse, and an asymmetric spatial-temporal module that captures both long-term and short-term action patterns, enabling efficient yet comprehensive modeling across the temporal context. Furthermore, we introduce an enhanced online inference pipeline that integrates cache-enabled attention to accelerate computation and a structured positional encoding strategy to support length extrapolation, allowing training on shorter sequences while leveraging longer temporal context during inference. This enhanced version, E2E-LOAD++, achieves superior speed-accuracy trade-offs and demonstrates strong potential for real-world deployment. Through comprehensive ablation studies and experiments, we validate the effectiveness and efficiency of our approach on three commonly used datasets: THUMOS-14, TVSeries, and HDD. The code is available at https://github.com/sqiangcao99/E2E-LOAD.