Validity and Reliability of the Observed Structured Point of Care Ultrasound Readiness Examination (OSPRE): A Pilot Study

Background In the context of competency-based medical education, more research is needed to define point-of-care ultrasound (POCUS) competency and validate learner assessment. To address noted gaps in the currently validated assessment instruments, we developed the Observed Structured POCUS Readiness Examination (OSPRE). The purpose of this study is to demonstrate evidence of validity and reliability of the OSPRE in Physician Assistant (PA) students. Methods A prospective cohort study was designed to demonstrate evidence of validity and reliability for the OSPRE instrument at one PA program. Students completed a hands-on skills assessment utilizing a simulated patient scenario involving a traumatic injury. Student performance was assessed in real time by one reviewer and asynchronously by a second reviewer blinded to the scoring of the initial reviewer. Inter-rater reliability was calculated for overall score and for each criterion of the OSPRE. Results Intraclass correlation coefficient for the final percentage score between the independent reviewers utilizing the OSPRE instrument was noted to be 0.86 (95% CI 0.56–0.95, p < 0.001), demonstrating overall good reliability. Level of agreement for inter-rater reliability for the specific individual criterion ranged between fair and substantial. Conclusion The OSPRE instrument provides a valid and reliable assessment of POCUS skills in PA students. When compared to previously validated assessment instruments, scores on the OSPRE had a wider range, suggesting that the OSPRE might have a greater discriminatory value for POCUS skillset.