An efficacy randomized controlled study of an AI-based assessment system in a governmental textbook approval process

This paper presents a randomized controlled study that evaluates the effectiveness of an Artificial Intelligence (AI)-based component designed to assess the technical quality of textbook images within the context of the Brazilian textbook program (PNLD). We adopted a parallel two-arm design with 1:1 randomization and included 76 textbook analysts. Participants completed a baseline test and a post-test after the initial assessment. We measured the primary outcome using post-test scores, while secondary outcomes evaluated analysts’ productivity and quality during textbook assessments. The current PNLD assessment process can take at least two years, involves hundreds of professionals performing manual tasks, and affects the entire educational system. One such task is assessing the technical quality of textbook images. To support this task, an AI-based system uses a convolutional neural network to solve a multiclass classification problem involving sharp, defocused-blurred, and motion-blurred images. The experiment showed that this specific AI component significantly increased productivity, while preserving quality, as the experimental group assessed substantially more images than the control group. Although the difference between pre-test and post-test results was modest, the findings indicate that the AI component can improve analysts’ ability to distinguish between different categories of image defects.