Real-world assessment (RWA): validity evidence for cataract surgery with a tool your faculty will actually use

Authors

Grace L. Paley, Department of Ophthalmology and Visual Sciences, Washington University School of Medicine, St. Louis, MO.
Hui Zhao, Kaiser Permanente, Sacramento, CA.
Victoria M. Addis, Scheie Eye Institute, University of Pennsylvania, Philadelphia, PA.
Casey J. Beal, Department of Ophthalmology, University of Florida Health, Gainesville, FL
Geetha K. Davis, Department of Ophthalmology, University of Missouri, Columbia, MO.
Gabriela M. Espinoza, Department of Ophthalmology, St. Louis University, St. Louis, MO.
Michael S. Lee, Department of Ophthalmology and Visual Neurosciences, University of Minnesota, Minneapolis, MN.
Stacy L. Pineles, Department of Ophthalmology, University of California Los Angeles, Los Angeles, CA.
Jamie B. Rosenberg, Department of Ophthalmology and Visual Sciences, Montefiore Medical Center, Albert Einstein College of Medicine, Bronx, NY.
Robert T. Swan, Department of Ophthalmology and Visual Science, SUNY Upstate Medical University, Syracuse, NY.
Misha F. Syed, Ophthalmology and Visual Science, University of Texas Medical Branch, Galveston, TX.
Evan L. Waxman, Department of Ophthalmology, University of Pittsburgh School of Medicine, Pittsburgh, PA.
Susan M. Culican, Department of Ophthalmology and Visual Neurosciences, University of Minnesota, Minneapolis, MN.Follow

Abstract

Background: We developed and implemented a real-world assessment tool (RWA) for resident surgery. It was designed to improve utility by reducing faculty and trainee cost (completion and training time, cognitive burden). RWA uses an entrustability scale where higher scores indicate a resident’s readiness for independent surgical practice. We describe a two-phase implementation of RWA and report validity evidence for cataract surgery assessment.

Methods: RWA was developed iteratively with resident and faculty input and piloted at a single residency program in phase 1. Phase 2 implemented RWA at 11 ophthalmology residency programs. The main outcome measures were 1) number of surgical assessments submitted and 2) correlation between entrustment score on resident cataract surgery assessments and time in training.

Results: After implementation of RWA, completed assessments increased for all participating programs. 1384 assessments of 111 residents by 112 faculty assessors were collected. Cross-sectional analysis of the assessments that examined cataract surgical procedural competency demonstrated that higher entrustment scores correlated with postgraduate year in training in both the single site (P=

Conclusions: RWA demonstrates low cost, high acceptability, construct validity and reliability for cataract surgical skills assessment. Thus, RWA has shown high utility for assessment of cataract surgery. Because it is procedure agnostic it can be used to assess resident surgical skill for any procedure. RWA provides useful data for residency programs to monitor trainee development.

Received Date

20-Feb-2024

Accepted Date

23-Aug-2024

Recommended Citation

Paley, Grace L.; Zhao, Hui; Addis, Victoria M.; Beal, Casey J.; Davis, Geetha K.; Espinoza, Gabriela M.; Lee, Michael S.; Pineles, Stacy L.; Rosenberg, Jamie B.; Swan, Robert T.; Syed, Misha F.; Waxman, Evan L.; and Culican, Susan M. (2024) "Real-world assessment (RWA): validity evidence for cataract surgery with a tool your faculty will actually use," Journal of Academic Ophthalmology: Vol. 16: Iss. 1, Article 10.
Available at: https://www.aupojournal.org/jao/vol16/iss1/10