A Semantic Knowledge-Integrated T5 Transformer Framework for Context-Aware Caption Generation and Behavioral Anomaly Analysis

The analysis of behavior anomaly is a key element of smart environments and intelligent surveillance, since in this case the interpretation of human behavior is unfeasible without semantic comprehension of the low-level motion features. Current anomaly detection methods are mostly based on handcrafted descriptors or statistical deviation models that are mostly non-contextual and non-semantic. In order to overcome them, this paper will suggest a unified design to consider the limitations, namely SAT5-BAA (Semantic-Aware Text-to-Text Transfer Transformer to Behavior Anomaly Analysis), a model that identifies anomalies based on the generation of semantically interpretable captions. The SAT5-BAA architecture has behavioral representations which are initially refined in a semantic enhancement module that combines contextual and temporal dependencies to enhance learning of discriminative features. A Flan-T5 encoder-decoder Transformer then processes the enriched representations to produce context-sensitive behavior captions of what is being done, what is being intended, and the surrounding situation as written natural language. An anomaly Analysis and prediction module that operates on Semantics is then used to detect abnormal patterns by measuring semantic differences between reference behavior captions and observed activity descriptions. Experiments on the UCF Anomaly Detection dataset have shown that SAT5-BAA is superior to classical feature-based and captioning baselines, achieving AUC-ROC improvements of 12.8% over MT-CMVAD, 10.4% over SwinAnomaly, and 9.6% over CVAD-GAN.