Utility of self-competency ratings during residency training in family medicine education-emerging countries: findings from Japan

Organized general/family medicine (FM) as a recognized medical specialty has a decades’ long history in multiple countries, dating back to the 1950s in the United Kingdom [1], Canada [2] and Australia [3] and to 1969 in the United States [4]. Still, in many countries, FM is unrecognized or just emerging as a specialty [5‐9]. In such places FM programs often lack resources for teaching and evaluating residents using the types of intensive one-on-one teaching procedures and complex competency or milestone evaluations that characterize FM education in more experienced countries.

FM programs may emerge because of policy changes that encourage or mandate their development. In this report, we refer to countries just beginning to develop systems or lacking official government-approved FM programs as Family Medicine Education Emerging (FMEE) countries. Emerging emphasizes that sociopolitical forces have reached a level where the determination across various constituencies to launch FM training is sufficient, relatively few faculty exist, and previous FM teaching experience is limited. FMEE academic programs face many barriers in launching a new discipline that impacts training at the student, resident, and faculty levels.

In Japan, a FMEE country, FM has not been recognized by the Japanese government as a medical specialty despite the efforts of advocates since the 1980s [10]. However, some recent crises have created a sociopolitical climate favoring FM development. Japan is a superaging society, and the rapid growth in the elderly population has highlighted significant problems: fragmentation of care; polypharmacy; need for after-hours care, including home and palliative care; and sky-rocketing health care costs.

In addition to the need for FM specialists [10] and status reports [11, 12], other literature emphasizes the potential role of FM in the community [13], academic medical centers [14], and especially to support women’s health [15]. Acknowledging Japan as a superaging society with fragmented medical care, the Japan Ministry of Health Labour and Welfare has articulated the clear need the need for general/FM physicians who can manage care efficiently, competently handle multiple medical problems, coordinate care, and address biopsychocial issues [16], and laid out plans for establishing general/FM as the 19th official specialty in Japan effective 2017 [17].

In light of new World Federation of Medical Education standards, the newly formed Japan Accreditation Council for Medical Education (JACME) will require medical schools to comply with new accreditation standards by 2023, including a newly required general medicine clinical rotation. Moreover, dedicated health tracks in medical schools focused on placing young physicians in health care shortage areas have seen limited success, despite lenient admission criteria, tuition waivers, and scholarships to individuals from health care shortage areas. These factors are forcing academicians to accept community and general medicine training as needed components of medical school training. The push from FM advocates and the crises confronting policymakers and academic leaders have provided the sociopolitical environment needed. In 2013, the Japanese government announced, as part of its restructuring of the specialist certification process, that a generalist specialty physician, called sogou shinryoui, will be recognized as the 19th Japanese medical specialty, starting in 2017 [18].

Despite national policy interest and academic medical institutions’ motivation to maintain international accreditation by offering clinical teaching in family/general medicine in Japan, some policy makers, academicians and subspecialists have been reluctant to change the status quo. Without previous experience working with or training family physicians, there may be apprehension as to whether FM residents can be trained across multiple disciplines and acquire the breadth of knowledge and skills required to treat the full spectrum of common conditions. Apprehension like this, which is not unique to Japan, challenges FMEE countries to provide evidence relatively quickly, despite limited resources, about the impact of FM training and to demonstrate programs’ ability to produce residents with the skills needed to provide competent womb-to-tomb care.

In medical residency training, competency is considered “an observable ability of a health professional, integrating multiple components such as knowledge, skills, values, and attitudes [19]. As a part of their Outcome Project, in 1999 the U.S. Accreditation Council for Graduate Medical Education (ACGME) defined six core competencies expected of medical residents: patient care, medical knowledge, interpersonal and communication skills, professionalism, and practice-based medicine [20]. Resident training programs must demonstrate that residents are developing these competencies by providing educational experiences and valid and reliable tools to use as evaluation instruments [21]. Along with contextualizing the concept of competency, the ACGME, together with the American Board of Medical Specialties, published a suggested “toolbox” of assessment methods, including Chart Stimulated Recall; objective structured clinical examinations (OSCEs), and 360° Global Ratings [22]. Recent movement surrounding resident assessment focuses on achievement of particular milestones [23]. Previous studies describe the creation of an evaluation instrument in their resident-training program, and analyze the results as indicative of the effectiveness of the instrument and the training program [24‐26].

In FM education-experienced environments, a gold-standard assessment would involve direct evaluation of a resident’s performance in patient care. A competency assessment would involve direct observation (e.g., Mini-CEX [27]) or review by a fully qualified family physician. Another option would be to examine board scores on a shelf examination each year (i.e., previously used test). Unfortunately, both procedures also require considerable infrastructure, including the availability of sufficient faculty time, shelf copies of a board examination (if one even exists) and a system to administer, score, and provide feedback to programs and residents. Unfortunately, both approaches are highly labor and cost intensive. A novel alternative would be a yearly resident self-competency evaluation. Using such an evaluation would provide a simple way of tracking individual development, and cumulatively, resident development in a program. This could thus be used to the extent newly developed FM residency training programs in FMEE countries are successfully teaching their residents the full breadth of care and providing reassurance that FM training program graduates with sufficient confidence in their knowledge and skills for FM practice.

In 2010 the University of Michigan Department of Family Medicine, along with three local hospitals, three mayors, and other supporters in Shizuoka Prefecture, Japan, received funding from the Prefectural Government to start a three-year FM training program unique in Japan for womb-to-tomb spectrum of care [28]. This research illustrates a longitudinal approach to self-competency assessments based on 5 years’ experience with the system in a new FM program in the FMEE country of Japan.

The study utilized a longitudinal cross-sectional self-competency assessment survey conducted yearly during 2011–2015. The survey was distributed using an online survey system (Qualtrics) each spring, after the academic year ended, and repeated yearly over four years. The University of Michigan Institutional Review Board and Shizuoka FM Program (SFM), Shizuoka, Japan, approved this educational research.

The research was conducted with SFM program residents. Nine responses were collected in 2011, 12 in 2012, 13 in 2013, 12 in 2014, and 9 in 2015. Residents were asked to complete the self-assessment survey online in the spring, at the beginning of each residency year, and on completion of the program. At matriculation into the program, 12 residents started at postgraduate year (PGY) 3; 4 at PGY4; 1 at PGY5; 1 at PGY 6; and 2 at PGY9. This variation occurs as there is not a matching system beyond preliminary training programs [29]. All SFM residents participated voluntarily. To avoid confusion with their PGY, residents were categorized by their FM training year (FMY), e.g., FMY1 means first year of FM training.

In conjunction with U.S. and Japanese faculty, a pilot paper survey with 92 conditions was tested in the inaugural SFM class, 2010. When we initiated the study there was no “list of common” conditions in primary care in Japan. Hence, we adapted a list for piloting based on 87 common conditions reflecting womb-to-tomb conditions commonly seen in acute care, chronic care, minor procedures, and preventive care/health promotion. We used as a reference common conditions recommended for medical students to learn about during their 4 week, third year clinical rotation at the University of Michigan, along with 5 emergency procedure protocols used commonly in Japan. Over the ensuing years, some edits for clarification, and additions and deletions, were made (further details available on request). Since 2014, the final instrument used for analysis contained 142 items: 30 items regarding acute care conditions, 28 on chronic care, 8 on women’s health, 38 on procedures (including three physical examination skills of the male/female genitalia/breast), 21 on health promotion (including newborn/young child, adolescent, and adult male/female), 12 on geriatrics and home care, and 5 emergency procedures protocols [basic life support (BLS), advanced cardiac life support (ACLS), pediatric advanced life support (PALS), advanced life support in obstetrics (ALSO), Japan advanced trauma evaluation and care (JATEC)]. Participants ranked their confidence for each domain or skill using a 100-point visual analog scale; 0 represents “can’t do at all” and 100 indicates “can do satisfactorily/completely.” Most respondents completed the survey in 10–15 min.

Completed surveys were downloaded from Qualtrics and imported into Microsoft Excel for organization before further analysis with Statistical Computer Software R. Means for individual residents and classes were calculated for each item and group to determine an average score per year and for the class as a whole. Mean scores for each category were compared by year. To compare progress over the residency, mean scores were plotted by FM residency program year (FMY) using box-plots.

Twenty residents completed one or more of the 59 surveys received over the 5 years of instrument administration. Eleven respondents were women; 9 were men. By Japanese regulation, all residents must complete a two-year preliminary rotating residency before entering an advanced residency such as the SFM program. As not all residents took the survey all years, the number of surveys taken differed by FMY; 16 trainees completed the survey at baseline (matriculation), 17 trainees responded after completing FMY1, 11 after FMY2, and 11 after FMY3. Four trainees completed the survey only once, another five completed two surveys, three completed three surveys, and eight completed four surveys. For privacy reasons, data on age and previous residency training were not collected.

These data suggest that the SFM program provides a training experience that produces physicians confident to practice the breadth of care needed by family physicians. The findings further illustrate that appropriately structured FM training programs can produce graduates prepared to provide the full spectrum of FM care. A resident self-competency assessment provides a simple and practical way to conduct resident self-assessments of skills, to monitor this over time, to use the data to inform residency program improvement, and to demonstrate the breadth of family medicine training to policymakers, and other stakeholders.