Impact of mcqs quality on the success rate of a qualifying exam for a postgraduate medical residency

Objective: To study items’ construction and psychometric analysis of the Specialty (SpX) Qualifying Residency Exam (QRE) of the Postgraduate Medical Institute “Y” (PGMY). Methods: A post-validation cross-sectional analytical study using a Non-Probability Purposive Judgmental sampling technique. The SpX was selected from the lowest three success rates of the 52 clinical specialties within the 2020-2023 QRE Cycles. Results: 175 candidates sat for QRE. The success rate was 10.86% (19). The QRE contained 120 A-type MCQs. Items without any flaws were 7 (5.8%). “Non-Vignette Stems” were 118 (98.3%); the majority of the “Lead-in items” were not in a question format. Two-thirds failed the “Cover-the-options” Test. 48 items (40%) had “Constructional Testwiseness” and “Irrelevant-MCQ” flaws. Furthermore, the mean Difficulty Index (DifI) was 45.9 ± 4.52, where 114 (86.7%) were within the acceptable. The Discrimination Index/Points Biserial (DisI/PBS) mean was 0.17± 0.02. 18 items (15%) had minus values. The Mean Distractor Efficiency (mDE) was 66.0% ± 0.09. Significant associations (p-value <0.05) were found between flaws and, the DifI and DisI/PBSDisI/PBS, HORST Index, and Bloom’s levels. Likewise, mDE showed a significant association with DifI but not with DisI/PBS. On the other hand, no significant association between the success rate and the MBBS curriculum style. Unlike the international trend of the same profession, the QRE had a zigzagging low success rate since cycle 2013. Conclusion: The items' quality significantly affected QRE. Other potentially influential factors deserve future multivariate analytical research. This consolidates the PGMY strategic plans for Exam Bank and Health Professions Education.