Development of a competency model for general practitioners after standardized residency training in China by a modified Delphi method

This was a study of developing competency model for general practitioners through a modified Delphi method. The Delphi method is a structured process for consensus-building among a diverse group of experts. The approach has commonly been adopted in medical research and remains today the most widely used method for selecting quality indicators in healthcare [21, 22]. The process ends when an agreement has been reached on the discussed topics. According to previous studies, two or three rounds are frequently used in the Delphi process [23, 24]. This study involved two rounds of questionnaires to an expert panel via e-mail from September to November 2020. All methods in the Delphi process were carried out in accordance with previous studies [23‐25] and research guideline for the Delphi survey technique [26].

This modified Delphi process was deployed based on two stages: (1) generating an initial set of relevant competencies derived from literature review, behavioral observation of GP–patient consultations, and critical incidents interviews of GPs; (2) conducting a 2-round, web-based Delphi survey of experts in general practice to prioritize and gain consensus on the essential competencies of GPs. Please see Fig. 1 for the process of the Delphi study.

A list of eligible experts was initially selected considering the representativeness of potential differences in background, occupational environment and clinical practices. The experts were invited upon the following criteria: (1) working as GP, researcher, or administrative leader in general practice department; (2) having at least 5 years’ working experience in general practice; (3) knowing about the work content of GP; (4) being familiar with “5 + 3” residency training; (5) being from various geographic regions within China. The participants were asked for their willingness to take part in the study. As in previous studies, a sample of at least fifteen has been suggested and more participants can increase the variety of expertise [27, 28]. Finally, 30 eligible experts were invited and 28 experts agreed to participate in this study.

Potential competencies were generated based on literature review, behavioral observations, and critical incidents interviews.

A preliminary list of competencies was constructed from three sources by literature review. Firstly, literature was searched in PubMed and three Chinese databases (China National Knowledge Infrastructure, Wanfang Data, VIP Chinese Periodical Services) with terms commonly used to describe GP (e.g., general practitioner, family physician, family doctor, community health worker), competency (e.g., competency, competencies, core competencies) and evaluation (e.g., evaluation, measurement, tool, indicator). A total of 37 published research papers describing domestic and foreign GPs’ competencies were identified form literature review (references of these 37 papers were shown at supplementary file 1). Secondly, 5 published competency models from international general practice organizations were also identified, including: the World Organization of Family Doctors (WONCA) [14], the College of Family Physicians of Canada (CFPC) [15], the Accreditation Council for Graduate Medical Education (ACGME) [16], the Royal College of General Practitioners (RCGP) [17], the Royal Australian College of General Practitioners (RACGP) [18]. Thirdly, 2 published policy documents of the content and requirement of GP residency training in China were reviewed [11, 29].

Potential competencies were extracted from these sources and screened by a panel of 2 reviewers (YW and FYW, Ph.D. candidates) according to the following criteria: (1) the indicators were used to measure the competency of GP; (2) the indicators were relevant to requirements of GPs’ work in China; (3) the indicators was relevant to the “5 + 3” residency training content. When there were doubts about whether an indicator should be retained, the research team would discuss together to make a decision. There were 88 competencies identified by the screening process.

Eleven GPs from 5 community health service institutions (CHSIs) in Beijing with stable amounts of patient visits were observed as a convenience sample. Each participating GP was observed when providing medical care in general practice consultations for one workday during November 2019 to January 2020. All consecutive patients visiting the GPs on the observing workday were recruited in the study with oral agreement. During the observation, patients’ reasons for encounter (RFEs) and medical services provided by GPs were recorded. Three research assistants (YW, FYW, and ZLP, Ph.D. candidates) were trained as observers before the observation. During the observation, the observers were seated in the least intrusive corner of consulting room and will not talk to the GPs and patients. There were 21 competencies related to GPs’ work content identified by the behavioral observation process.

The same 11 GPs as in behavioral observation were invited, 8 GPs participate in the critical incidents interview and 3 GPs declined due to other work or family affairs. During the interview, participants were asked to describe incidents with good effect and incidents with bad effect. Questions were asked based the “STAR Principle”, which included ‘What kind of situation was it at that time?’ (Situation), ‘What was the main task you faced at that time?’ (Task), ‘In that incident, what skills did the you display?’ (Action), ‘What was the final result of this incident?’ (Result). The information from incident interview was taped, transcribed, and coded. Three researchers (YW, FYW, and ZLP, Ph.D. candidates) extracted the information about GPs’ competencies from the incident interview data respectively. When there were doubts about whether a description of competency should be retained, the research team would discuss together to make a decision. There were 35 competencies related to GPs’ work content identified by the critical incidents interview process.

A total of 144 competencies were identified by these three processes above. After deleting duplicate competencies and integrating the competencies with similar dimensions, a preliminary list of 63 potential competencies were constructed. Then, the competencies were discussed in detail one by one in a research team meeting (2 educators and 3 GPs), concentrating on whether these competencies were measurable and wording them by referring to other competency models. The competencies considered as unmeasurable by more than half of the participants were removed. After further removal and integration, 46 potential competencies were left, which were categorized into 7 domains.

All 46 potential competencies were formatted into Delphi questionnaire. Importance and feasibility of the competencies were rated on a 1–9 Likert scale (1 = not important/feasible; 9 = very important/feasible). Spaces were left for experts to make comments on these existing competencies or recommend new competencies which they considered should be included in.

Descriptive analysis was used to describe the characteristics of participates and results. Means [with standard deviation (SD)] were used to report continuous variables, while frequencies (%) were used to report categorical variables. The Data management and analysis were performed using Statistical Package for Social Science (SPSS), version 22.0.

All of the 28 experts participated in both two rounds of Delphi survey. Among them, 21 experts were from Beijing, 2 experts were from Shanghai, and other 5 experts were from 5 provinces of China (Hainan, Zhejiang, Anhui, Hebei, Inner Mongolia). There were more female participants (64.3%) in this panel, and the mean age of the experts was 47.9 years (standard deviation: 9.3 years). Nearly one third of the participants were GPs in CHSI, 39.3% were GPs in hospital, 10.7% were researchers in primary care, and 17.9% were leaders of CHSI. The average length of general practice experience was 14.4 years, with 67.9% experts working for over 10 years in this field. There were 78.6% experts had master or PhD degree and 64.3% experts were with senior grade title (Table 1).

In the first round, 44 (95.7%) of the original 46 competencies achieved consensus in round one. The median score of importance and feasibility ranged from 8.00 to 9.00 and 7.00 to 9.00, respectively. The percentage of panel ratings in the top tertile (7–9) for importance and feasibility ranged from 85.7 to 100% and 69.9 to 100%, respectively. There were 2 competencies did not achieve 75.0% agreement in terms of feasibility, which were “3.2 Screen of at risk individuals for population health issues” (69.9% agreement) and “5.5 Allocate external resources of the institution for optimal patient care” (71.4% agreement). The indicator “3.4 Provide chemoprevention care” was recommended to be removed by 2 experts (a GP and a researcher), despite the agreement being achieved. So, three competencies were deleted in the first round.

Description of two domains were modified. “2. Basic Medical Services” was modified into “2. Patient care”. “3. Preventive care and basic public health service” was modified into “3. Basic public health service”. Eight competencies (1.10, 2.3, 2.7, 2.8, 3.7, 4.1, 4.5, and 5.4) were modified based on the experts’ comments. For example, “1.10 Record electronic health information” was modified into “1.10 Use electronic health record system effectively”. New competencies were suggested by 13 of the 28 experts in the first round. Only 7 new competencies were suggested by more than two experts and hence included in the second round (Table 2). Thus, 50 competencies were included in the second round.

At this step, 50 competencies were evaluated, including retained, modified, and new competencies. In the second round, the median score of importance and feasibility ranged from 7.50 to 9.00 and 7.00 to 9.00, respectively. The percentage of panel ratings in the top tertile (7–9) for importance and feasibility ranged from 85.7 to 100% and 82.1 to 100%, respectively. As a result, more than 75% of the experts gave ratings in the top tertile (7–9) to 50 competencies, all of which had a median of 7 or above and a high degree of consensus was achieved in terms of importance and feasibility. Descriptive statistics including the median and percentage agreement for each indicator is shown in Table 2.

At the end of the Delphi process, 50 competencies finally achieved consensus in the second round in 7 domains: knowledge and skills (10 competencies), patient care (9 competencies), basic public health services (6 competencies), communication (7 competencies), teamwork (6 competencies), professionalism (6 competencies) and education, learning and research (6 competencies) (Table 3).

This study was a rigorous process, which involved a multi-method approach to analyze behaviors associated with the performance of GP, including literature review, behavioral observation of GP–patient consultation, and critical incidents interview of GP. Then a modified Delphi survey was conducted with 28 general practice experts to achieve consensus on the most essential competencies of GPs after standardized residency training in China. The final consensus set included 50 competencies categorized into 7 domains.

As described in foreign competency models, the competency of GP involved many aspects, such as patient care, communication, professional knowledge and skills, professionalism and practice-based learning [14‐18], which were also important for GPs’ role in China. As the first contact for patients within PHC system, the main task of GP is patient care in clinics, including medical care to patients with acute and chronic health problems. Professional knowledge and skills are the foundation of patient care. Effective communication was crucial to doctor-patient relationship [32], which was also indicated by the GPs in critical incidents interview in our study. For general practice, communication and empathy are essential in patient-centered care [33]. In addition, practice-based learning plays a very important role in the improvement of GPs ability, as the evidence or knowledge are clinically relevant and reflect the circumstances of real practice [34]. In the competency model for GPs after standardized residency training in China, the competencies mentioned above were identified as 34 items based on literature, GPs’ work content in China, and the experts’ consensus.

It is notable that there are three aspects special in China: basic public health service, teamwork, and research. To tackle health inequity, providing universal basic public health services for residents is the main goal of the new health reform in China. Since 2009, basic public health service programs have been widely carried out across PHC sectors in China [35], which now includes 14 categories, such as health records management for residents, health education, vaccination, reporting of infectious diseases and public health emergencies, and etc. [36]. GPs in PHC system play key roles in delivering majority of the basic public health services. Therefore, we made an attempt to identify competencies in this domain on the basis of basic public health services programs and the experts’ consensus. Six competencies were identified in an independent domain “3. Basic public health service”. Since 2011, many provinces and cities in China have explored the model of “family doctor contract” services, which help to let patients have their personal doctors and improve the continuity of care based on GP team with a GP, a nurse, and a preventive care physician [1, 37]. The ability of working effectively with others in a collaborative team-based model is emphasized to GPs in China. Besides, collaborative leadership is also an important indicator as GP plays the role of leader in the team. In China, research ability and paper are critical to physicians in career advancement. During the standardized training program, GPs can be trained with the ability of scientific research [11], which should be evaluated. Therefore, three domains of competencies were included in our competency model.

Compared with previous assessment tools of GPs’ competency in China, the competency model for GPs after standardized residency training in this study made improvement on method and content. A literature review of 31 studies evaluating competencies of GPs in China, conducted by our research team before this research, found that more than half of the included studies did not use a psychometrically robust, high-quality instrument to measure the competency of GP [20]. In this study, the competency model for GPs after standardized residency training was developed through a modified Delphi method. The Delphi questionnaire encompassed previous studies regarding GP competencies and published competency models from international general practice organizations [14‐18]. In addition, further competencies related to the work content of GPs were added by behavioral observation of GP–patient consultations and critical incidents interview of GPs [25]. These two methods had not been introduced in the process of developing competency assessment tools for Chinese GP before. Besides, for the “5 + 3” residency training model, the evaluation of GP trainees usually focused on process assessment to monitor the progress of training. In this study, we developed a competency model for evaluation of GPs after standardized residency training. In questionnaire preparation, the Content and Rules for Standardized Training of General Practitioners (2019 revised edition) was referred [11]. In Delphi process, the experts were familiar with “5 + 3” residency training. Therefore, this competency model can be used to explore the performance and competency of GPs after standardized residency training in workplace. The results may provide feedback to GPs and trainers of the “5 + 3” residency training model for further improvement.

In this study, the importance and feasibility of the competencies achieved consensus in the Delphi process. This set of competencies provided a basis for competency measurement of GPs after standardized residency training in China, which still needs to be tested in practice in further studies [38]. We suggest compiling these competencies in our model into a questionnaire for self-evaluation by GPs or multi-source assessment by other staff and patients in workplace [39].

The quality of panel experts and their opinions on given topics is seen as key factor of the Delphi technique [40]. In this study, the presence of different professionals (GPs in hospitals, GPs in CHSIs, leaders of CHSIs, and researchers in universities) and geographical areas (28 experts from 7 provinces of China), along with the average length of general practice experience (14.4 years) and similar ratio of experts with GPs in actual practice (64.3% female experts VS 58.0% female GPs in actual practice [6]), suggested that our expert panel represented a broad and experienced group. Furthermore, the response rate of our study was 100% in two rounds of Delphi process. This was a satisfactory result as response rate was a recognized problem in Delphi study. Importantly, this meant that experts had strong interest and active participation in this topic.

There are limitations of this study. First, although experts in this study were from different geographical areas, most of them were from Beijing and the proportion of experts in other provinces was low. They may not adequately represent the full spectrum of views held by individuals in different regions across China. Second, patients’ opinion was not involved in the sources of competency. Patients are the customers and beneficiaries of general practice services, who may give deep insights in health care experience and doctor-patient communication [25]. This should be taken into consideration in further study. The methodology of Delphi process relies on the perception of experts, which may entail further evidence from implementation in real practice settings [41]. Further study is needed to apply this competency model and confirm the validity of these competencies.