Human or AI? An Interpretable Method for Auditing Free-Text Data in Online Psychological Research

Online psychological studies increasingly rely on free-text responses, but the availability of generative AI creates new challenges for evaluating the authenticity and integrity of data pertaining to human introspection. We present an interpretable text-analysis workflow for identifying linguistic cues associated with perceived AI authorship in digital journal entries. Across a two-week journalling study, 251 participants submitted 2,808 text entries, which were independently labelled by human annotators as human-written, AI-generated or invalid, and compared with classifications from a commercial AI-detection tool. We extracted 127 transparent features capturing length, semantic coherence, stylistic consistency, readability, natural-language-inference-based consistency, lexical overlap and Linguistic Inquiry Word Count 2022 psycholinguistic categories. On a high-confidence subset defined by convergence between human judgements and automated detection, interpretable classifiers strongly distinguished entries perceived as human from those perceived as AI-generated. A decision tree achieved 95.3% accuracy using a small set of auditable rules based on verb rate, syllable count, article use and sentence count, while a linear SVM identified 18.6% of entries as ambiguous cases suitable for human review. Entries perceived as AI-generated were longer, more coherent, more stylistically uniform and more formally structured, whereas entries perceived as human showed greater present focus, self-reference and stylistic variability. These findings provide a transparent framework for auditing possible AI mediation in remote psychological text data, while showing how interpretable models can support human review rather than replace it.