Competency-based Resident Education

Otolaryngol Clin N Am 40 (2007) 1215–1225

Competency-Based Resident Education Bradley F. Marple, MD Department of Otolaryngology/Head and Neck Surgery, University of Texas, Southwestern Medical Center at Dallas, 5323 Harry Hines Boulevard, Dallas, TX 75390-9035, USA

Medical education in the United States has evolved over the course of the last century from that of an apprentice-based system to its current state of organization through a series of stepwise changes. Early organizational efforts implemented by pioneers such as William Halstead [1] and Abraham Flexner (Flexner report, 1910) around the turn of the century revealed the poor state of medical education at the time, and set into motion a series of expectations and resultant improvements that ultimately impacted both medical education and public safety. The current energy surrounding the ACGME Outcomes Project and its impact on transition to a competency-based resident training paradigm are the latest steps in the evolution of medical education.

Brief history of the Accreditation Council for Graduate Medical Education/role of the Residency Review Committees The Accreditation Council for Graduate Medical Education (ACGME) was established in 1981 to serve as a national governing body charged with accreditation of post-MD residency training programs within the United States. In effect, the fiduciary responsibility of the ACGME is to the public as demonstrated in its mission statement: ‘‘We improve health care by assessing and advancing the quality of resident physicians’ education through accreditation.’’ In its role as a national governing body, the ACGME sets general policy and the direction of resident education. However, recognizing that significant differences exist between specialties, oversight at the specialty level is relegated to 27 specialty-specific committees. It is the responsibility of these Residency Review Committees (RRCs) to periodically review every residency training program to ensure programmatic compliance with the standards that are set forth by the ACGME.

E-mail address: [email protected] 0030-6665/07/$ - see front matter Ó 2007 Elsevier Inc. All rights reserved. doi:10.1016/j.otc.2007.07.003

oto.theclinics.com

1216

MARPLE

ACGME Outcomes Project In February 1999, the ACGME endorsed the Outcomes Project, which is funded by the Robert Wood Johnson Foundation and designed to focus on educational outcomes [1]. This project would eventually lead to fundamental changes in the way residents are educated. Accreditation would no longer be granted on the basis of a residency program’s potential to educate trainees as demonstrated by process assessment (process-oriented education). Instead, accreditation would be granted based on a series of measurable outcomes intended to demonstrate an individual resident’s ability to provide effective patient care, as well as the residency program’s ability to provide such training (outcomes-oriented education). With the Outcomes Project in mind, the ACGME introduced a new model of accreditation in July 2002 that was based on two major requirements. First and foremost, it was assumed that the effectiveness of resident education would be more efficient if alterations were made in resident lifestyle that decreased fatigue. The first requirement, therefore, addressed a reduction of the resident workweek to a maximum of 80 hours. In many cases, this requirement caused a significant decrease in the number of hours logged by residents during training. The resultant impact on workflow, in turn, resulted in major changes to the residency process. Education, communication, and transfer of patient care between health care providers, along with other traditional processes of training programs, were modified, thus leading to the development of novel training strategies. Although it was initially feared that this potential for reduction in resident exposure to patient care would lead to physicians ill equipped to function effectively following graduation [2,3], it appears that more effective educational processes and less fatigued residents have largely overcome these concerns [4]. On the other hand, limiting the resident workweek has effectively resulted in measurable decreases in clinical experience during residency training [5,6]. Resolving the mutual demands of a balancing of reduced patient care exposure with an optimization of educational opportunities has given rise to novel approaches to resident education [7]. In light of the concerns about the ACGME requirement limiting resident duty hours, the second requirement was appropriately focused on improvements in the quality of the resident educational experience [8]. Under the model proposed by the ACGME, accreditation would be based on a series of outcomes-based measures. This approach has created a need to refine reasonable goals and objectives specific to each program, to develop ways of effectively capturing and optimizing available clinical experiences, to develop a series of reasonable and validated measures of outcome, and to develop processes by which collected data would facilitate improvement in resident and residency program performance. Further, development of the tools necessary to teach trainees to implement a problem-based approach to medical care should serve to develop a culture of lifelong learning.

COMPETENCY-BASED RESIDENT EDUCATION

1217

To date, implementation of the Outcomes Project has been the responsibility of the individual programs, with guidance by the ACGME and the individual RRCs. Timeline of the ACGME Outcomes Project The Outcomes Project is a long-term initiative by which the ACGME is increasing emphasis on educational outcomes as the basis for accreditation of residency education programs [8]. In contrast to the current practice of accreditation based on a potential to educate as measured by determining compliance with a set of existing requirements, the goal of the Outcomes Project is to emphasize the actual accomplishments of a training program as assessed by measurable outcomes. Stated differently, it intends to measure how effectively education in each program impacts its trainees’ abilities. The Outcomes Project has been implemented in phases that were proposed on a strict timeline [9]. In each phase a set of programmatic focuses and accreditation focuses were listed. Under the guidance of the RRC program evaluation process, each program would receive feedback regarding progress. Phase 1 (July 2001 through June 2002) required that an initial response be formed to the changes in ACGME requirements. Specifically, each program was required to define specific objectives that would allow residents to demonstrate learning in the six general competencies. Additionally, the general competencies were to be integrated into resident didactic and clinical educational experiences. In Phase 2 (July 2002 through June 2006), the focus and definition of the competencies and assessment tools used to evaluate outcomes were to be sharpened. Learning opportunities in all six of the competency domains were to be demonstrated. The evaluation processes were to be improved as needed by each program, under the guidance of the RRC. In addition, aggregated resident performance data would be accumulated for each program’s Graduate Medical Education (GME) Committee internal review. In Phase 3 (July 2006 through June 2011), each program is to fully integrate the general competencies and their assessment into both learning and clinical care. Resident performance data are to be used as the basis for improvement and to provide evidence for the program’s accreditation review. In essence, this enables Phase 3 of the implementation of the Outcomes Project to be the point at which competency-based training is to become operational. Finally, in Phase 4 (July 2011), the general competencies and their assessment are to be expanded to develop models of excellence. In this phase, as in the preceding phases, site review by the RRCs will provide an opportunity to identify and disseminate models of excellence developed within individual programs. In theory, this process will lead to identification of benchmarks that will serve to facilitate continual improvement.

1218

MARPLE

The six general competencies The first major activity of the Outcomes Project was the development of the six general competencies for residency training, which were endorsed by the ACGME in 1999 [10]. This process of development was stimulated by an introspective assessment of how adequately physicians in the current model of residency training were prepared to practice medicine in the arena of the changing health care delivery system. The general competencies were derived through evaluation of existing research on qualities necessary for physicians to function effectively. Further input was gathered from a variety of GME stakeholders. The ACGME approved specific language regarding the general competencies and evaluation processes in September 1999 [10]. For the purposes of accuracy, the description of the ACGME general competencies used is that approved by the ACGME [11]. The six general competencies are 1. 2. 3. 4. 5. 6.

Patient care Medical knowledge Practice-based learning and improvement Interpersonal and communication skills Professionalism Systems-based practice

Patient care Residents must provide patient care that is compassionate, appropriate, and effective for the treatment of health problems and the promotion of health. Residents are expected to  Communicate effectively and demonstrate caring and respectful behaviors when interacting with patients and their families  Gather essential and accurate information about their patients  Make informed decisions about diagnostic and therapeutic interventions based on patient information and preferences, up-to-date scientific evidence, and clinical judgment  Develop and carry out patient management plans  Counsel and educate patients and their families  Use information technology to support patient care decisions and patient education  Perform competently all medical and invasive procedures considered essential for the area of practice  Provide health care services aimed at preventing health problems or maintaining health  Work with health care professionals, including those from other disciplines, to provide patient-focused care

COMPETENCY-BASED RESIDENT EDUCATION

1219

Medical knowledge Residents must demonstrate knowledge about established and evolving biomedical, clinical, and cognate (eg, epidemiological and social-behavioral) sciences and the application of this knowledge to patient care. Residents are expected to  Demonstrate an investigatory and analytic thinking approach to clinical situations  Know and apply the basic and clinically supportive sciences that are appropriate to their discipline Practice-based learning and improvement Residents must investigate and evaluate their patient care practices, appraise and assimilate scientific evidence, and improve their patient care practices. Residents are expected to  Analyze practice experience and perform practice-based improvement activities using a systematic methodology  Locate, appraise, and assimilate evidence from scientific studies related to their patients’ health problems  Obtain and use information about their own population of patients and the larger population from which their patients are drawn  Apply knowledge of study designs and statistical methods to the appraisal of clinical studies and other information on diagnostic and therapeutic effectiveness  Use information technology to manage information, access on-line medical information, and support their own education  Facilitate the learning of students and other health care professionals Interpersonal and communication skills Residents must demonstrate interpersonal and communication skills that result in effective information exchange and teaming with patients, their patients families, and professional associates. Residents are expected to  Create and sustain a therapeutic and ethically sound relationship with patients  Use effective listening skills and elicit and provide information using effective nonverbal, explanatory, questioning, and writing skills  Work effectively with others as a member or leader of a health care team or other professional group Professionalism Residents must demonstrate a commitment to performing professional responsibilities, adherence to ethical principles, and sensitivity to a diverse patient population. Residents are expected to demonstrate  Respect, compassion, and integrity; a responsiveness to the needs of patients and society that supercedes self-interest; accountability to

1220

MARPLE

patients, society, and the profession; and a commitment to excellence and ongoing professional development  Commitment to ethical principles pertaining to provision or withholding of clinical care, confidentiality of patient information, informed consent, and business practices  Sensitivity and responsiveness to patients’ culture, age, gender, and disabilities Systems-based practice Residents must demonstrate an awareness of and responsiveness to the larger context and system of health care and the ability to effectively call on system resources to provide care that is of optimal value. Residents are expected to  Understand how their patient care and other professional practices affect other health care professionals, the health care organization, and the larger society and how these elements of the system affect their own practice  Know how types of medical practice and delivery systems differ from one another, including methods of controlling health care costs and allocating resources  Practice cost-effective health care and resource allocation that does not compromise quality of care  Advocate for quality patient care and assist patients in dealing with system complexities  Know how to partner with health care managers and health care providers to assess, coordinate, and improve health care and know how these activities can affect system performance Components of a competency-based education program In essence, the current Outcomes Project is an example of a competencybased educational model. This model is designed to focus on the performance of the individual trainee, as measured in outcomes, as he or she works to master educational goals and objectives. For this form of education to be effective, a number of elements must be in place. Further, these elements should be integrated in such a way that meaningful data are generated and continually integrated into the educational environment, thus providing a progressive, graduated experience for the learner. A clear set of educational goals and objectives serves to set the stage for the educational environment and aligns the expectations of the learners with those of the program. Goals and objectives should be specific to the educational site and experience level of the resident, in effect serving to continually challenge and develop trainees’ skill sets. It is important that this set of goals and objectives can be objectively measured to track progress and identify areas that need remediation. Finally, the goals and objectives should attempt to reflect the need to gain experience in all six of the general competencies.

COMPETENCY-BASED RESIDENT EDUCATION

1221

At first glance, the efforts required to transform residency programs as suggested by the ACGME Outcomes Project appeared monumental, but most programs have been able to modify existing didactic and clinical experiences to align with the general competencies required by the ACGME [12,13]. A recommended approach to help assess programmatic alignment with new ACGME regulations is to address three questions: Do the residents achieve the learning objectives set forth by the program? What evidence can the program provide that it does so? How does the program demonstrate continuous improvement in its educational process? [1]. Measurement, tracking, and documentation of the goals and objects set forth by each program provide a mechanism by which to ensure effective education. However, this step in the process of educational transition may also present a challenge. Large amounts of data generated by this process will require secure storage and organization if they are to be useful. Many programs have used computer-based systems that can log resident duty hours, track a variety of performance measures, and identify areas of noncompliance/substandard performance. Further, these systems can prepare reports (ie, performance ‘‘dashboards’’) and alert program directors early in the course of deviations from expected performance [1]. Varying levels of access can be provided to residents, faculty, program directors, and chairmen ‘‘chairpersons’’ (or simply ‘‘others’’)? to allow for ongoing access to pertinent information. A variety of assessment tools has been developed and made available as outcome metrics through the ACGME Outcomes Project Toolbox [8]. Table 1 is a partial list of assessment tools that are currently in common use. Each of the listed metrics has been made available for use by residency programs. Ultimately, programs are given the flexibility to use the assessment tools made available by the ACGME to facilitate measurement of the effectiveness of a residency program in its preparation of residents. Although some assessment tools may be tailored to assess most, if not all, of the general competencies, some are less capable of this function. Thus, the use of several tools is recommended to optimize the validity of the assessment portfolio collected for each resident. Given that each program may approach integration of common program requirements, goals and objectives, and the way in which data are gathered in a different fashion, some general guidelines are offered by the ACGME. Programs are expected to [8]  Provide learning opportunities in each general competency domain  Use multiple and overlapping assessment methods  Aggregate data such that they improve the educational program

Expectations for the future As noted previously, the ACGME Outcomes Project is intended to facilitate a change in the current system of medical education. Fundamental to

1222

Table 1 Common methods of assessment Description

360 evaluation instrument

360 evaluations are measurement tools designed to be completed by a variety of individuals involved in the examinee’s sphere of influence. Evaluations are completed in a ‘‘360 ’’ fashion, signifying participation by peers, superiors, and subordinates. A standard survey or questionnaire is used to gather information and can be tailored to easily align with the general competencies. A variety of rating scales can be used. A global rating is a retrospective assessment in which a rater assesses general categories of an ability (eg, patient care skills, medical knowledge, interpersonal and communication skills). The rating is based on general information and impressions derived from multiple sources of information (eg, direct observations or interactions; input from other faculty, residents, or patients; review of work products or written materials) and is collected over a period of time (eg, end of a clinical rotation). Rating forms frequently allow the evaluator to make scaled responses (eg, 5 ¼ outstanding, 4 ¼ good, 3 ¼ fair, 2 ¼ marginal, 1 ¼ unsatisfactory). Written comments allow evaluators to expand on responses. Procedure, operative, or case logs document patient encounters, surgeries, or procedures performed. The logs include counts of cases, operations, or procedures. This type of information is intended to document sufficient exposure to clinical experience.

Global rating of live or recorded performance

Procedure, operative, or case logs

MARPLE

Assessment tool

Patient surveys

Simulations and models

COMPETENCY-BASED RESIDENT EDUCATION

Portfolios

Patient surveys assess satisfaction of the health care experience from the perspective of the patient and frequently include comments about the physician’s care. The questions often assess satisfaction with physician communication skills, time spent with the patient, perceived competency, and so forth. Patients are typically asked to rate satisfaction through the use of rating categories ranging from excellent to poor. Each rating is given a value that can then be averaged to generate a comprehensive score overall or separated into separate clinical domains. A portfolio is a collection of resident-collected products that provides evidence of learning and achievement related to a learning plan. This portfolio may contain written documents, photographs, case logs, and other forms of information. Resident preparation of these materials provides an important opportunity to reflect on what he or she had learned. Additionally, a written synopsis of the portfolio provided by the resident can be used to identify areas of strength and weakness. Simulations use models that duplicate actual clinical scenarios and are used to assess clinical skills and performance. Effective simulations allow examinees to perform skills or act out potentially harmful situations without placing a patient at risk. Additionally, simulations can be used to provide a safe environment within which a skill can be practiced and acquired before its introduction into the patient care setting. Feedback can be immediate, allowing the learner to modify mistakes and reinforce successes in subsequent simulations. (continued on next page)

1223

1224

Table 1 (continued ) Description

Standardized ral examination

The standardized oral examination uses a realistic patient case as the basis of a series of questions administered by a trained physician examiner. Questions assess acquisition of clinical information, rationale for requesting testing, interpretation of findings, and treatment plans. A written examination is composed of multiple-choice questions created to sample medical knowledge. Each question consists of an introductory statement (or question), which is followed by a series (typically four or five) options. The examinee selects one of the options as the correct response. Checklist evaluations use a series of ‘‘yes’’ or ‘‘no’’ questions addressing specific behaviors, activities, or steps that make up a more complex competency or competency component.

Written examination

Checklist evaluation

MARPLE

Assessment tool

COMPETENCY-BASED RESIDENT EDUCATION

1225

this change is a migration from the current process-based model to one of a competency-based model in hopes of generating data on which continuous refinements can be made. Effective use of these data, once processes are in place, can enable individualized training that is tailored to reinforce strengths and remediate problems early in the training process. This change has the potential to result in modifications of the residency accreditation process for residencies as well as the certification process for trainees. As an example, case logs emphasizing absolute numbers of operative cases may be replaced, or supplemented, by outcome metrics that demonstrate an individual resident’s proficiency in the skills necessary to effectively accomplish a range of related procedures. Simultaneously, a program is provided with continuous information that enables improvements in the methods by which training takes place. Making use of such an example of competency-based training may provide for individual certification based on demonstrated proficiency.

References [1] Joyner BD. An historical review of graduate medical education and a protocol of accreditation Council for Graduate Medical Education compliance. J Urol 2004;179:34–9. [2] Winslow ER, Bowman MC, Klingensmith ME. Surgeon workhours in the era of limited resident workhours. J Am Coll Surg 2004;198(1):111–7. [3] Romano M. Hours of doctors-in-training: who’s counting? ACGME’s restrictions limiting workhours denounced as inadequate. Mod Healthc 2002;32(33):18–9. [4] deVirgillo C, Yaghoubian A, Lewis RJ, et al. The 80-hour workweek does not adversely affect patient outcomes of resident education. Curr Surg 2006;63(6):435–9. [5] Weatherby BA, Rudd JN, Ervin TB, et al. The effect of resident work hour regulations on orthopaedic surgical education. J Surg Orthop Adv 2007;16(1):19–22. [6] Short AC, Rogers SJ, Magann EF, et al. The 80-hour workweek restriction: How are OB/GYN resident procedure numbers affected? J Matern Fetal Neonatal Med 2006; 19(12):801–6. [7] Woodrow SI, Segouin C, Armbruster J, et al. Duty hours reforms in the United States, France, and Canada: is it time to refocus our attention on education? Acad Med 2006; 81(12):1045–51. [8] Accreditation Council on Graduate Medical Education. Available at: http://www.acgme. org. Accessed May, 2007. [9] Accreditation Council on Graduate Medical Education. Available at: www.acgme.org/ outcome/project/timeline/TIMELINE_index_frame.html. Accessed May, 2007. [10] Accreditation Council on Graduate Medical Education. Available at: http://www.acgme. org/outcome/project/OPintrorev1_7–05.ppt-12. Accessed May, 2007. [11] Accreditation Council on Graduate Medical Education. Available at: www.acgme.org/ outcome/comp/compFull.asp. Accessed May, 2007. [12] Dyre PL, Strauss RW, Rinnert S. Systems-based practice: the sixth core competency. Acad Emerg Med 2002;9:1270–7. [13] Hobgood CO, Riviello RJ, Jouriles N, et al. Assessment of communication and interpersonal skills competencies. Acad Emerg Med 2002;9:1305–9.