Zidong Zhou, Cheng Liu, Xuanhe Wang
Although RT did not materially improve overall score-category classification, it improved the prioritization of provisional decisions at risk of high-confidence disagreement (HCD) with complete-questionnaire categories. RT-based calibration also modestly reduced calibration error relative to the response-only kernel baseline, while standard isotonic calibration performed best. Among 14,576 scale-level test decisions, 579 met the operational definition of HCD. The confidence-plus-RT model achieved an average precision of 0.772, compared with 0.540 for the confidence-only model and 0.491 for the shuffled-RT control. At a 5% participant-escalation capacity, the confidence-plus-RT model captured 30.6% of HCD decisions, compared with 15.2% using confidence alone; at 10% capacity, the corresponding capture rates were 56.1% and 32.8%. These capacities corresponded to average questionnaire burdens of 19.9 and 20.8 items per participant, respectively, compared with 37 items under universal completion. The HCD-ranking advantage was observed across alternative confidence thresholds and item-selection designs, with the largest gain for PSS-14.
INTRODUCTION: Reduced-item mental health symptom questionnaires can reduce respondent burden in large-scale screening, but some provisional results may still require confirmation through completion of the remaining items. We examined whether item-level response time (RT) could improve selective escalation after a 19-item screen.
METHODS: Using fixed-order data from 24,292 students who completed the PHQ-9, GAD-7, PSS-14, and ISI with item-level RTs, we conducted an offline participant-specific adaptive replay. Complete-questionnaire score categories served as the reference for evaluating the provisional categories issued after the 19-item screen.
RESULTS: Although RT did not materially improve overall score-category classification, it improved the prioritization of provisional decisions at risk of high-confidence disagreement (HCD) with complete-questionnaire categories. RT-based calibration also modestly reduced calibration error relative to the response-only kernel baseline, while standard isotonic calibration performed best. Among 14,576 scale-level test decisions, 579 met the operational definition of HCD. The confidence-plus-RT model achieved an average precision of 0.772, compared with 0.540 for the confidence-only model and 0.491 for the shuffled-RT control. At a 5% participant-escalation capacity, the confidence-plus-RT model captured 30.6% of HCD decisions, compared with 15.2% using confidence alone; at 10% capacity, the corresponding capture rates were 56.1% and 32.8%. These capacities corresponded to average questionnaire burdens of 19.9 and 20.8 items per participant, respectively, compared with 37 items under universal completion. The HCD-ranking advantage was observed across alternative confidence thresholds and item-selection designs, with the largest gain for PSS-14.
DISCUSSION: The results show that RT can serve as a complementary response-process signal for directing limited full-questionnaire completion capacity toward provisional decisions with the highest estimated HCD risk. Prospective, questionnaire-specific studies should evaluate this workflow in independent samples and live screening settings.