Bridging the evidence gap: A methodological roadmap for evaluating chatbots as psychological change agents

The use of artificial intelligence is rapidly advancing. However, despite substantial technological progress and ongoing interest in conversational agents transporting psychologically-informed interventions for well-being, behavior change, and personal development, empirical evidence on the effectiveness and psychological impact of chatbots designed to facilitate such non-clinical, goal-oriented change still is in a nascent stage, limiting cumulative knowledge building. This article outlines a methodological roadmap for evaluating the effects of human-chatbot interaction as a means for psychological change while drawing on established evaluation traditions in psychotherapy, training evaluation, and counseling research. Using AI-supported coaching as an illustrative case, it proposes feasible study designs to assess overall effectiveness and identify active intervention ingredients, presents selected outcome measures progressing from proximal reactions to indicators of real-world change, and discusses evaluation approaches eventually culminating in multi-channel fusion. By introducing a complementary evaluation level for psychologically-informed conversational agents extending established HCI evaluation frameworks, this paper aims at contributing to enhanced comparability of future evaluation studies and a harmonized evaluation science for chatbots designed to produce measurable psychological change.