Summary by Joe Tauro, M.D.
OLC Chair
Some Definitions
- PBT: Proficiency based training
- A general approach to teaching that involves teaching and follow up that measures proficiency
- PBP: Proficiency based progression
- It’s the same as PBT but there are interim proficiency levels which must be assessed and passed before moving on to the next level
- Validity:
- The PBT has been shown to test what you want to test (Face Validity)
- Is complete and comprehensive (Context Validity)
- Can tell the difference between novices and experts (Construct Validity)
- Translates into better performance in real life (Transfer Validity)
- Achieving validity in PBT is very important but not easy. It is critical especially if such a program is adopted for high stakes testing like a board exam
Tools we are currently using and improving
- PBT/PBP arthroscopic knot tying
- PBT/PBP "FAST"
- Fundamentals of Arthroscopic Surgery Training. This is a basic trainer that teaches triangulation, scope and instrument maneuvering and is capable of simple model-based repairs of the labrum and rotator cuff. We are also working on a VR simulation augmented version of FAST.
- PBT/PBP "ASSET"
- Arthroscopic Surgical Skills Assessment Tool. ASSET has been developed to test proficiency in hip, ankle, knee and shoulder. It involves specific steps which must be completed with hard stops and remediation if necessary until completed. It uses a Likert scale for measurement which is 1-5. 1 is bad, 5 is expert. It construct validity but because it has a 1-5 scale for each step it is more prone to interpretation errors than a binary (yes/no) scoring system like Copernicus. Greg Nicandri (this year’s Snyder Teaching Award winner) helped develop ASSET and tells me that it also has face and construct validity but I’m not sure it does based on my review of the published papers.
- Copernicus PBT/PBP
- The critical difference is that scoring is binary and thus more accurate. Also has face, content and construct validity. In regards to teaching, it is the only methodology (to my knowledge anyway) that has been shown to translate into better performance (Transfer Validity). It is probably the best teaching and testing tool available but has proven difficult to validate for the many procedures we need it for.
- Virtual Reality Simulator (VRS) PBT/PBP
- We have signed a 2-year development deal with VirtaMed in which they provide the simulators, software and hardware engineers and some of the data collection and analysis personnel and we provide the expertise for curriculum development.
- VRS has the huge advantage of being reproducible, allowing unlimited repetition to acquire skills and the ability to gather performance data and assess proficiency automatically
What are we doing now?
- PBT Knot tying, FAST and ASSET have been incorporated in all of the Foundations (Residents) Courses at the OLC. It has gone very well so far but we continue to improve on implementation and data collection
- Copernicus: The first implementation of Copernicus Bankart repair has been incorporated into the June Fellows Course. The whole Masters team has done a great job with this. Dr. Rick Angelo, one of the prime developers of Copernicus, will be there to assist. What we learn in June, Copernicus programs will continue to be blended into our offerings at the OLC as long as we have more procedures to incorporate.
- VR Simulation
- We have assembled knee, shoulder, hip and FAST teams and are working with VirtaMed on Curriculum development. We expect FAST, knee, shoulder basic curriculum and testing to be done this year. Hip may take until 1st quarter 2019. In 2019 we will work on more advanced procedures like ACL, RCR, labral repair for shoulder and hip.
- We hope to start Copernicus incorporation into the simulators in late 2019.