Quantification of practice effects in a self-administered online cognitive pre-screening test

Abstract Background With growing prevalence of mild cognitive impairment and dementia, online self-administered pre-screening tests hold the promise to more objectively identify individuals who stand to benefit from prioritized access to supervised cognitive testing and in-depth neurological evaluation, while reducing the burden of unnecessary tests for physicians and the health care system. Objective We aimed to quantify practice effects in same-day and mid-term test-retest applications of the XpressO cognitive pre-screening test, to develop novel methods to stratify practice effects in the absence of ground truth, to quantify the discrimination power in retests, and to incorporate information from practice effects to improve the discrimination power for retests. Methodology We performed a retrospective observational study to analyse data from a large, self-enrolled online cohort. We created descriptive statistics for test-retest comparisons of all relevant performance metrics, developed an unsupervised machine learning approach to impute the cognitive status of participants, and used this to compare the discrimination power in the initial test-retests on the same day or after a gap of two weeks or more. Results Significant practice effects were found in a self-enrolled online population, in same-day retests and after several weeks, with differentially larger effects observed for presumably cognitively impaired participants. Practice effects reduced the discrimination power of the original test score in the retests. Conclusion Whenever participants can perform unlimited, uncontrolled, and unsupervised practice sessions, increased task variation is desirable for future cognitive pre-screening tests. Quantified practice effects contain independent, additional predictive information regarding the cognitive status of online participants.