John Obeid, Christopher Bobier, Alex R Gillham, Adam Omelianchuk, Daniel J. Hurst
INTRODUCTION: Integrating artificial intelligence (AI) into medical education poses barriers and opportunities for medical educators. One of those opportunities is content creation for medical ethics assessment. Assessing the performance of AI in generating multiple choice questions (MCQs) in ethics that aligns with United States Medical Licensing Examination (USMLE) is important. The present study evaluates the performance of eight AI models-GPT-4 Turbo, GPT-3.5 Turbo 0125, o1 Mini, o1 Preview, GPT-4, Claude 3.5 Sonnet, Claude 3 Opus, and Gemini-in regard to the relevance, clarity, and accuracy of generated ethics-based MCQs. METHODS: Each of the eight models was tasked with generating two sets of question, answer, and explanation for each of 13 selected USMLE-aligned student learning outcomes related to medical ethics. Responses were rated by four independent experts in medical ethics on relevance, clarity, and accuracy. Performance metrics were assessed using mean, median, range, and total scores, with an overall percentage score computed for each model. Qualitative responses noting strengths and weaknesses were collected. RESULTS: Claude 3.5 Sonnet had the highest overall performance (86.28%), accuracy (85.57%), and clarity (91.73%). o1 Mini followed closely with an overall score of 85.19%, while GPT-4 had the highest rating for relevance (88.65%) and a total score of 84.87%. GPT-4 Turbo and o1 Preview had the lowest overall score at 76.79%. Identified weaknesses included incorrect answer selection, best answer not available, and the question was not ethics-based. CONCLUSION: The findings indicate clear potential for medical educators tasked with designing medical ethics MCQs for teaching and assessment. However, expert oversight is needed to ensure proper utilization. DATA AVAILABILITY STATEMENT: Data supporting the findings of this study are available from the corresponding author on request.