Competency Based Education: Quality over Quantity

On August 22^nd 2016, the American Board of Surgery (ABS) announced the newly defined category minimum numbers approved by the RRC-Surgery for residents graduating in the 2017-2018 academic year.¹ An increase in the minimum number of total operative procedures from 750 to 850 procedures was approved. There was also an increase in the minimum number of chief resident procedures from 150 to 200, along with an increase in the cases in surgical critical care from 25 to 40. There were no specified areas of focus for the increase volume (i.e. more breast surgeries), nor was a specific reason for this change articulated. Why was such a decision made, and what is driving this increase in case volume?

One potential reason could be the perception that residents are graduating unprepared. In 2013, Mattar and colleagues published their survey results of Fellowship Program Directors asked to assess the level of preparedness and skill in the incoming class.²  The results were surprising.  A total of 21% arrived unprepared for the operating room, 38% demonstrated a lack of patient ownership, and 66% were deemed unable to operate for 30 unsupervised minutes of a major procedure. Up to 28% were unfamiliar with therapeutic options and 24% were unable to recognize early signs of complications. Furthermore, a survey conducted of graduating general surgery residents on operative confidence in 2014 demonstrated low levels of individual confidence in common bile duct exploration (18%), groin lymph node dissection (33%) and control of a liver injury (51%).³ Of the 16 different surgical procedures reported, only basic laparoscopy and open inguinal hernia repair received a level of confidence above 90%. There is no data to support that increasing the case volume throughout residency and during one’s chief year is the answer to the issues discussed above. Rather than increasing the quantity of unspecified cases, we should be increasing the quality of teaching both clinical and procedural skills. From there, we should define surgical expectations clearly and develop comprehensive assessments to demonstrate a resident’s ability to perform a particular operation.

How do we go about demonstrating a resident’s competency today?  The ACGME has broadly defined 6 clinical competencies for all graduating residents: medical knowledge, patient care, professionalism, interpersonal and communication skills, problem based learning and improvement, and systems based practice.⁴ From there, eight domains of surgical practice were established as part of the General Surgery Milestone Project as a joint initiative between the ACGME and the ABS in 2014: care for disease and conditions, performance of operations and procedures, performance of assignments and administrative tasks, maintenance of physical and emotional health, teaching, improvement of care, coordination of care and self-directed learning.⁵ Within the 8 practice domains, there are 16 sub-competencies for which residents are expected to be assessed based on the 6 core competencies. An example of a sub-competency includes a resident’s ability to perform an efficient and accurate initial history and physical for patients admitted to the hospital.  This sub-competency encompasses the core competency of Patient Care, along with the practice domain of Care for Diseases and Conditions.  There is no specific assessment tool associated with such a task. With this particular model in existence today, how are we able to know if a particular resident can assess a specific disease process and ultimately perform the correct operation independently?

For the 2017-2018 academic year, there are 13 categories for case types with a minimum number of cases assigned in each category. The categories are quite broad and encompass many different operations. For instance, in the “skin, soft tissue and breast” category, the minimum number is now 65, as opposed to 25. A resident could do 65 breast surgeries or 65 melanoma operations and still meet the acceptable rate in this category for board eligibility. There are no requirements for case diversity.  The mere number of cases is not and should not be as important as the type of cases within each category and the skill with which these cases are being performed. Another issue is the fact that there is not a list with the truly essential operations a general surgeon should perform. The SCORE website has 132 cases listed as core surgical procedures. Included are operations such as a salping-oophorectomy and digital nerve block. Is this list what all general surgeons should be trained to do? At what level should a resident be able to do all of these cases? If the expectation is that graduating residents are able to perform these operations independently, then we need the ability to formally assess all 132 cases.

Are we losing sight of the goal of surgical residency? In general, the purpose of the 5 years of training is to teach one the knowledge and skill to treat common surgical problems independently. Should one be interested in specialization beyond that, then a fellowship year would be recommended. As such, those graduating should be able to perform core operations autonomously based on their practice and exposure during training.

We therefore need to move away from shear volume and truly assess the quality and autonomy with which the resident is performing the operation. Each resident will become proficient by doing a different number of cases per operation. Some may require 5-10 surgeries, while others 15-20. By generalizing the case volume per category, one’s individual learning and ability to become competent is not taken into consideration. We need to develop a system to assess residents on specific operations. Not only should the surgical case be assessed, but pre-operative preparation and post-operative care should also be a standard part of the evaluation process. Just because a resident can perform a case does not mean he or she has the appropriate knowledge to properly diagnose and treat the illness leading to the operation. Our current evaluation system does not explicitly factor this important part of surgical care into the requirements.

In my opinion, there are two main issues that exist. The first is that we have to define what operations all general surgeons should be competent in. The second is that we need to propose how to prove a resident is competent to treat a disease process independently. One suggestion, for which the ABS is currently exploring, is the development of surgical Entrustable Professional Activities (EPAs) with associated assessments. An EPA is an activity that a clinician at various levels of training is expected to be able to perform independently. Thirteen Core EPAs have now been developed for medical students entering residency by the Association of American Medical Colleges (AAMC).⁶ Examples include recommending and interpreting common diagnostic and screening tests, and entering and discussing orders and prescriptions. Several residency programs, including pediatrics and psychiatry, have also worked to define EPAs for their professions. Overall, this model has been proposed for use in medical domains, along with suggestions for implementation.^{7, 8} To our knowledge, EPAs based on surgical procedures do not presently exist in a validated form. By outlining specifically, per operation, the requirements for a general surgeon, a framework could then be developed for a system that could assess the comprehensive treatment of common surgical diagnoses.

Once we agree upon the specific surgical diseases, along with the methods for assessing a resident’s competency in these conditions, what will the next steps be? Leaders in various domains of surgery and surgical education will be challenged to decide what skills and abilities are essential for residents to be independently competent in order to treat common diseases requiring surgical intervention.  By this, we mean that residents have acquired the skills, ability, and knowledge to treat a disease process and that these components have been clearly defined and assessed so that we can be assured that graduating surgeons can treat these diseases independently.^{9, 10} This will require numerous discussions amongst leaders about competency and what we expect from graduating surgical residents. Additionally, we will need more multi-institutional development and testing of assessment tools in order to ensure that the evaluations we are utilizing are fair, valid, and reliable.

In conclusion, in light of the updated operative requirements in 2016, the ABS is undoubtedly concerned about the training general surgery residents are obtaining today. However, a focus on numbers of procedures performed by residents will not get at the majority of issues that exist with surgical training today. As a group, we need to first define which procedures are essential for those trained in a general surgery residency to be able to perform independently. Once determined, we will then need to agree upon the definition of competence to be able to create comprehensive surgical EPAs and associated assessments. This will provide the framework for defining what is clearly required of all general surgeons upon completion of training. We will then have the ability to be fully confident in the disease-specific skills and knowledge of our graduating residents.

References:

1. General Surgery resident performance assessments [The American Board of Surgery web site]. Available at: http://www.absurgery.org/default.jsp?certgsqe_resassess. Accessed September 26, 2016.
2. Mattar SG, Alseidi AA, Jones DB, Jeyarajah DR, Swanstrom LL, Aye RW, Wexner SD, Martinez JM, Ross SB, Awad MM, Franklin ME. General surgery residency inadequately prepares trainees for fellowship: results of a survey of fellowship program directors. Ann Surg. 2013; 258(3): 440-449.
3. Fonseca AL, Reddy V, Longo WE, Udelsman R, Gusberg RJ. Operative confidence of graduating surgery residents: a training challenge in a changing environment. Am J Surg. 2014; 207(5): 797-805.
4. Kavic MS. Competency and the Six Core Competencies. JSLS: Journal of the Society of Laparoendoscopic Surgeons. 2002;6(2):95-97.
5. ACGME and ABS. The General Surgery Milestone Project [ACGME web site]. July 2015. Available at: http://www.acgme.org/portals/0/pdfs/milestones/surgerymilestones.pdf. Accessed September 26, 2016.
6. Association of American Medical Colleges. Core Entrustable Professional Activities for Entering Residency [AAMC web site]. 2014. Available at: http://members.aamc.org/eweb/upload/Core%20EPA%20Faculty%20and%20Learner%20Guide.pdf. Accessed on September 26, 2016.
7. ten Cate O, Chen HC, Hoff RG, Peters H, Bok H, van der Schaaf M. Curriculum development for the workplace using entrustable professional activities (EPAs): AMEE guide no. 99. Medical Teacher. 2015; 37(11): 983-1002.
8. ten Cate O, Hart D, Ankel F, Busari J, Englander R, Glasgow N, Holmboe E, Iobst W, Lovell E, Snell LS, Touchie C. Entrustment decision making in clinical training. Academic Medicine. 2016 Feb 1;91(2):191-8.
9. Carraccio C, Englander R, Van Melle E, ten Cate O, Lockyer J, Chan MK, Frank JR, Snell LS. Advancing Competency-Based Medical Education: A Charter for Clinician–Educators. Academic Medicine. 2016; 91(5): 645-9.
10. Alman BA, Ferguson P, Kraemer W, Nousiainen MT, Reznick RK. Competency-based education: a new model for teaching orthopaedics. Instructional course lectures. 2012; 62: 565-569.