Unpacking SoTL “Failure”: Lessons from a Fellowship Program Investigating Generative AI’s Impacts

The experience of failure in designing, implementing, interpreting, and/or disseminating Scholarship of Teaching and Learning (SoTL) research is not new to educational developers; it is, however, a relatively new focus for extended inquiry and exploration. The word failure may evoke complicated emotions or assumptions, but normalizing and learning from SoTL failure helps educational developers and instructors refine and elevate their SoTL practice. This paper contributes to the growing literature on SoTL failure by exploring project-level SoTL failure in the context of a fellowship program implemented by a Center for Teaching and Learning (CTL). This fellowship is designed to support novice SoTL researchers from all disciplines in evaluating the causal impact of technology-enhanced learning (TEL) interventions, specifically generative AI use, on student learning outcomes. Defined within the context of this fellowship, a failed project is one that did not collect at least one direct measure of student learning, did not analyze (or could not effectively interpret) the data regarding implications for future course design and delivery, or did not disseminate findings. We identify multiple sites for potential SoTL project failure within the processes of study design, implementation, and dissemination and consider how TEL interventions present further potential complications. Our goal is to share lessons learned through projects that failed or nearly failed as well as the preventative and mitigating strategies and scaffolding we developed to address the most common causes of failure. Image Credit: Judy Brooks