SmartSeg: A non-parametric approach for wearable camera video temporal segmentation

Wearable cameras provide an efficient and convenient way to record our lives, supporting real-time documentation and analysis across various domains. Recent research has explored diverse methods for temporal segmentation, which aim to transform unstructured video data into structured events. This transformation facilitates deeper video understanding, optimizes computational resources, and improves the accessibility and interpretability of video content for both machines and humans. However, unlike conventional videos, wearable camera recordings present unique challenges. These include highly unstable camera perspectives, diverse activities across various environments, and flexible duration. As a result, traditional temporal segmentation methods often fail to return effective results. This paper introduces SmartSeg, an unsupervised, non-parametric approach for segmenting wearable camera videos without labeled data. By capturing the fundamental meanings of the video, SmartSeg aggregates the video through the Temporal Self-Similarity Metric encoder and groups sequences of frames into coherent events through clustering techniques. We evaluated SmartSeg on three diverse datasets. We achieved a 50% increase in Mean-over-Frames(MoF) compared to the state-of-the-art on one egocentric dataset. We conducted a real-world case study on nursing simulations, demonstrating SmartSeg’s ability to effectively segment complex, noisy interactions with diverse activity transitions. The results highlight SmartSeg’s robustness in handling long, unstructured, and visually challenging wearable camera videos, establishing it as a promising tool for real-world video temporal segmentation tasks.