Education Indicators for Internal Medicine Point-of-Care Ultrasound: a Consensus Report from the Canadian Internal Medicine Ultrasound (CIMUS) Group

With increasing evidence and support in using point-of-care ultrasound (POCUS) at the bedside,^1‐4 its application in internal medicine is gaining traction in North America and internationally.^4‐7 In 2018, the American College of Physicians issued an official statement in support of POCUS use for internal medicine.⁴ Similarly, for the practice of hospital medicine, the Society of Hospital Medicine has issued a position statement providing guidance for hospitalists and administrators regarding application, training, assessment, and program management for POCUS.⁸ Internal medicine residency training programs have only recently begun to incorporate POCUS in their curricula. A national survey in 2013 revealed that only 25% of internal medicine residency programs in the USA offered a formal POCUS curriculum.⁹ Since then, a number of programs across the USA have described successful efforts at introducing POCUS to their internal medicine training programs, both in a workshop format^10,11 and longitudinal curricula.^12‐14

Despite these advances, internal medicine point-of-care ultrasound (IM POCUS) curriculum development and implementation continues to be a challenge globally for many residency training programs. For example, a survey study in Chicago suggests that learners continue to feel incompetent in the use of ultrasound,¹⁵ and learners in Canada similarly reported low level of IM POCUS skills.¹⁶ Barriers to IM POCUS education consistently cited in the literature include lack of access to equipment, lack of established curricula, limited availability of educational time, and lack of trained faculty.^9,17‐20 Introducing a novel technology such as POCUS into clinical practice requires significant resources and new infrastructure (e.g., ultrasound machines, image archiving systems), and relies on a limited supply of professionals with expertise. As such, integrating POCUS is expected to be formidable.²¹ Potential solutions to integrate POCUS, therefore, must be engineered to anticipate and overcome these obstacles—a multifaceted approach is necessary.

On a global scale, education is diverse and heterogeneous. To allow for the comparisons of the state of education worldwide, the Organization for Economic Co-operation and Development (OECD) publishes annual results on education indicators.²² These indicators characterize education outputs, financial and human resources invested, access to education, and learning environments.²² These process, structure, and outcome measures provide timely and quantifiable key information metrics for policy decision-makers and can assist in ensuring quality across programs.²³

Education indicators are rarely utilized in medical education, given the relative stability in medical education over the past decades.²⁴ The introduction of POCUS has, in many ways, produced a technological disruption that is challenging in the current era of stability²⁵ and is raising new issues in the geographically diverse landscape of Canada. Establishing standards to define program processes, structure, and outcomes is a crucial step to ensure that POCUS program development is deployed in a thoughtful manner and with broad support. Education indicators provide policy makers and educators such as hospital administrators, program directors, and POCUS faculty with a clear and instructive framework to guide curriculum development, implementation, evaluation, and monitoring efforts. Establishing standardized measures of quality can help advance POCUS education in a number of ways.²⁶ First, POCUS educators can improve the design and delivery of their POCUS curriculum by adhering to quality metrics espoused by the education indicators. Second, education indicators can assist policy makers such as hospital administrators and program directors in where to direct necessary resources. Third, by adhering to education indicators, greater uniformity in quality can be achieved across programs. This study seeks to establish consensus-based recommendations for education program indicators for internal medicine POCUS training in Canada.

The Canadian Internal Medicine Ultrasound (CIMUS) group is composed of members who are leaders across Canadian internal medicine residency programs, with POCUS and/or education expertise.²⁷ This group previously developed consensus-based recommendations for the components of an internal medicine POCUS curriculum.²⁷

In this follow-up project, the CIMUS group held a 4-h consensus meeting concurrent with the proceedings of the Canadian Society of Internal Medicine Annual Meeting in Toronto, ON, on November 4, 2017. The objective was to establish consensus recommendations on IM POCUS education indicators. Members met in person or via teleconference. Two members are representatives from the Royal College of Physicians and Surgeons of Canada’s specialty committees in Internal Medicine and General Internal Medicine, but participated as individuals, rather than as representatives of the specialty committees. The Royal College of Physicians and Surgeons of Canada is the standard setting body for Canadian residency programs. In addition, two members, both general internists with expertise in quality improvement (AA and SR), were invited to provide expert input for quality indicator development processes and frameworks. The meeting was facilitated by one POCUS expert with expertise in medical education and consensus methods (IM), and one internist with expertise in quality improvement (AA).

During the meeting, information regarding education indicators for IM POCUS programs, their definition, purposes, and examples of historical and contemporary uses were presented and discussed.^23,28‐31 Specifically, participants were introduced to the indicator framework of inputs, processes, and outputs.^23,32 This framework approximates the general standards categories that are currently endorsed by the Royal College of Physicians and Surgeons of Canada, the College of Family Physicians of Canada, and the Collège des Médecins du Québec for evaluation and accreditation of Canadian residency programs.³³

A refresher of the modified nominal group technique for establishing consensus was provided to the meeting participants.³⁴ We planned to conduct no more than three rounds of voting (first round in-person plus no more than two rounds online). All rounds were conducted anonymously and the same participants were invited to vote. Prior to voting, the voting members informally agreed upon characteristics of indicators that were beyond the scope of this study because of poor feasibility, impracticality, or low level of evaluation. Specifically, our group agreed at this time to exclude indicators relating to the quality of specific metrics (e.g., quality of ultrasound machines, quality of teachers, and quality of their teaching) due to low feasibility, indicators relating to patient outcomes due to current impracticality and low feasibility (see Supplementary Online Appendix 1). We also excluded indicators related to learner satisfaction due to low level of evaluation based on the Kirkpatrick four levels of training evaluation.³⁵ Specifically, indicators that capture only learner reaction were not considered.

A total of 22 members participated in the meeting, representing 13 Canadian academic institutions across 7 provinces (Table 1). At the initial meeting, 14 members participated in-person and 8 via teleconferencing.

Based on the results of our consensus, we recommend that 22 education indicators (Table 2) be considered in the development, implementation, evaluation, and monitoring of IM POCUS training curricula for Canadian internal medicine residency programs. These indicators cover domains including teaching, learning environment and program organization, data management and quality assurance, and assessment and program evaluation. These indicators may serve three overarching purposes: directing curriculum development and implementation efforts, benchmarking curriculum progress over time, and allowing for cross-comparisons and standardization of performances across programs. While only 22 indicators are recommended, it is important to keep in mind that these represent a core number of elements that programs should consider essential to track during implementation and longitudinally; other indicators that did not achieve consensus may also be important to consider.

While our recommended education indicators address a variety of curricular elements, the majority of the recommended indicators relate to learning environment and program organization. Some of these indicators are similar to training and quality assurance processes recommended by policy statements and guidelines from other official bodies.^37‐40 However, our recommendations differ from these in three ways. First, while these other policy statements provide general guidance for educators and training programs, to our knowledge, ours is the first group to recommend actual indicators. These indicators provide quantifiable measures that residency programs can target during POCUS curriculum implementation. Second, existing guidelines from other associations are directed towards specialties where the practice of POCUS has now been fully integrated and its scope well defined.^37,39 Therefore, explicit guidance and ongoing program monitoring may not be as critical for these fields as their training pathways are already well established. In contrast, IM POCUS is a relatively new field. Explicit guidance from indicators may be more valuable. Lastly, because our indicators were developed by consensus of representatives from the majority of Canadian internal medicine residency programs, the involvement of key stakeholders may help programs across the country produce a more uniform educational landscape.

There are several limitations to our study. First, our group is composed entirely of Canadian educators who are familiar with the current enablers and barriers in the Canadian IM POCUS education system. These indicators may not be generalizable to other settings and may change over time. For example, in settings where educational efforts are directed in a more top-down approach,⁵ the current recommended indicator regarding support from the internal medicine residency program director may be of lesser importance than support from national certifying bodies. Further, Canada only has 17 internal medicine residency training programs. Stipulating uniform use of indicators on a national level for countries with higher number of training programs may be more challenging. Second, from the outset, our group recognized a number of limitations in the scope of the education indicators. The quality of some indicators (e.g., quality of the ultrasound machines, quality of the didactic teaching) may be difficult to capture, given the subjective nature of these measures as well as a lack of available metrics. As such, we acknowledge that our consensus list is not comprehensive. Third is the issue of representation; despite involving national and local internal medicine POCUS leading educators in this study, we do not have provincial ministry and health authority involvement. From an educational perspective, involving only proximal representatives is a limitation. In addition, given that POCUS is a relatively new skill, a number of our experts have limited experience in teaching POCUS. However, the development of education indicators requires expertise in educational principles. To that end, our panel is deliberately diverse to include medical education experts. Fourth, the time frame between the first and second round was rather long. While recommendations on consensus studies do not specify time limits between rounds,^34,36 the longer this time frame, the longer it would presumably take participants to re-acquaint themselves with the questions,⁴¹ which may lower intra-rater reliability. Last, operationalization of indicators will need to be better defined. For example, while we intend for indicator no. 5 (total hours of faculty time spent) to encompass total hours of faculty spent on curriculum development as well as delivery, whether or not this indicator can be accurately and feasibly captured must first be established with additional studies. Precise definitions of each indicator will then need to be iteratively established.

The Canadian Internal Medicine Ultrasound (CIMUS) group recommends that these 22 education indicators be used to guide and monitor internal medicine POCUS curriculum development, implementation, evaluation, and monitoring efforts in Canada.