Evaluating the concept retrieval technique as an Assessment Tool: Generalizability and Concurrent Validity Evidence

Background In health professions education, assessing students’ conceptual understanding and knowledge retrieval is essential for effective clinical practice. This study aimed to examine the generalizability and concurrent validity of the concept retrieval technique (CRT) as an alternative to open-ended examinations and to determine whether differential weighting of concepts enhances its concurrent validity. Method A quantitative study conducted at a private university in United Arab Emirates. Two studies were conducted. Study 1 was an experimental study conducted with 41 first-year Bachelor of Science in nursing (BSN) students. Students studied short texts on two topics. Target word lists of key concepts were prepared for each topic, with each concept assigned a weight reflecting its relative importance. To assess the reliability of the concept retrieval technique (CRT), Generalizability theory (G theory) was employed. Concurrent validity was evaluated by comparing CRT performance with essay tests (memory retrieval) and multiple-choice questions (concept recognition). The three sets of exams were administered concurrently. First, CRT reliability was computed using the G theory and D study. Second, correlations between CRT and essay test scores were calculated to determine the concurrent validity of CRT. Third, the relationship between retrieval and recognition outcomes was examined by comparing CRT and essay test scores with MCQs. Fourth, evaluation of weighted versus unweighted scoring to examine whether weighting improved concurrent validity. Study 2, reassessed CRT concurrent validity in a different nursing domain (mental health nursing course) among 43 third- and fourth-year BSN students. Results The concept retrieval technique demonstrated good generalizability across raters and domains (G = 0.87). The D-study indicated that the number of domains, rather than raters, was the primary determinant of reliability, with two domains and a single rater sufficient to achieve good reliability. CRT also showed considerable concurrent validity with essay-based assessments. Comparisons between weighted and unweighted concept grading revealed minimal differences in student performance, with no significant effect on the concurrent validity of the CRT. Conclusions CRT is a reliable and unbiased assessment method, with good generalizability across Raters and domains. It also has a notable concurrent validity when compared to essay-based evaluations. Unweighted scoring of CRTs is favored for its simplicity and time efficiency.