Fairness of a Professionalism-Oriented Situational Judgment Test for Residency Selection in the United States: A Predictive Bias Analysis by Gender, URiM Status, and International Medical Graduate Status.

Residency selection has traditionally emphasized cognitive measures such as Unites States Medical Licensing Examination (USMLE) Step 1, Step 2 Clinical Knowledge (CK), and clerkship grades, even though physician performance encompasses cognitive and noncognitive competencies. Situational judgment tests (SJTs) have emerged as a promising complement, demonstrating criterion-related validity and producing smaller subgroup score differences than cognitive measures. However, smaller mean differences do not establish fairness. The Standards for Educational and Psychological Testing distinguish between mean group differences and predictive bias. The latter occurs when an assessment systematically over- or underpredicts outcomes for certain groups regardless of true construct standing. Whether residency-focused SJTs demonstrate predictive bias has not been examined-an important gap, as unknown bias risks unfairness to applicants and could undermine efforts to build a more representative physician workforce. This multiyear study investigated whether a professionalism-oriented SJT demonstrated predictive bias by gender, underrepresented in medicine (URiM) status, or international medical graduate (IMG) status relative to Accreditation Council for Graduate Medical Education (ACGME) milestone performance. Using moderated multiple regression, we tested for intercept differences (predicted outcomes differ between groups at the same SJT score) and slope differences (the strength of the predictor-outcome relationship differs by group). The study included 24 residency programs and three fellowships at a single academic medical center. Predictive bias analyses included 422 residents; mean score analyses included 2,680 applicants; and background characteristic analyses used diversity survey responses from 1,659 interns. The SJT predicted year-end milestone performance across all subgroups (B = 0.14-0.17, p < 0.05), with no intercept or slope differences by gender, URiM status, or IMG status, indicating no evidence of predictive bias. Gender differences across selection measures were negligible (d = -0.08 to 0.23). URiM applicants scored lower than non-URiM peers on all measures, with larger gaps on cognitively loaded exams (Step 1: d = 0.74; Step 2 CK: d = 0.66) than on the SJT (d = 0.30) or milestones (d = 0.28). IMG applicants scored lower on the SJT (d = 0.45) and Step 2 CK (d = 0.29) but not Step 1 (d = -0.06). URiM interns reported greater structural disadvantage; IMGs were markedly less likely to report native English proficiency. This study provides initial empirical evidence that a professionalism-oriented SJT does not exhibit predictive bias by gender, URiM status, or IMG status in relation to residency milestone performance. These findings constitute a preliminary step toward a fairness evidence base for residency-focused SJTs, pending replication in applicant populations and operational selection settings. The SJT's smaller subgroup gaps relative to USMLE exams suggest value in holistic admissions, though this inference requires validation in samples not restricted by prior selection.