Standard Setting for Dental Knowledge Tests: Insights from Judge Feedback and Focus Groups

Introduction Criterion-referenced standard setting is commonly employed to establish passing marks in healthcare professional examinations, its application in Malaysian dental schools is limited. This study aimed to examine the judges’ feedback and thought processes when utilising the modified Angoff and Ebel methods in the final-year multidisciplinary exam, involving two panels of judges at a dental faculty in Malaysia. Methods A convergent mixed-methods approach was used, combining quantitative feedback questionnaires and qualitative focus group discussions (FGDs) over a three-day standard setting workshop at The National University of Malaysia. One-best answer items and short answer questions were standardised using the modified Angoff and Ebel methods. The questionnaires evaluated the judges’ understanding, confidence and preference for the defensibility of the standard setting methods. The FGD explored the judges’ experiences and thought processes, analysed through inductive thematic analysis. Results Twelve judges from various specialties participated. The feedback questionnaires showed no significant differences between the modified Angoff and Ebel methods or between Groups A and B. The judges reported enhanced understanding and confidence levels from Rounds 1 to 3. Judges’ preferences for each method varied. The themes included the following: confidence in standard setting, continuous vs. category scoring, the role of diverse expertise in decision-making, alignment with course learning outcomes, barriers to effective standard setting and the impact of training as a catalyst for effective standard setting. Conclusion Groups A and B shared similar feedback, experiences, and perspectives regarding the standard setting activities. The modified Angoff method was straightforward and easy; it gave judges more control, but it provided a subjective framework, relying on intuition and experience in setting the Angoff ratings. In contrast, the Ebel method offered a structured, objective framework, but judges felt they had less control, found the process more time-consuming, and some referred to the probability that borderline candidates would answer correctly within each category in Ebel grid to determine the difficulty level. These findings provide insights into standard setting in dental education and emphasis the importance of structured training to enhance procedural validity.