Learning health systems in primary care: a systematic scoping review

We included data sources that described the implementation or plans for developing an LHS within a primary care setting. We defined an LHS as an organization or a network, which rapidly analyzes health data while incorporating best-practice guidelines to directly feedback to and improve clinical care (either through clinical research, quality improvement or decision support tools). In this case, clinical research included comparative effectiveness analyses, prediction modelling, pragmatic trials, or other big data analytics where the findings could be readily applied in practice. We excluded research networks where only conventional knowledge translation through academic publications was occurring without more direct feedback of the knowledge to improve care. We included integrated health systems or networks that also had inpatient care, or care in other settings, if they included primary care in their health system or network. We did not include pediatric LHSs, since this is considered specialty care in some countries including Canada. We only included studies published after the IOM report in 2007 [6], and we restricted the search to English language articles only. Finally, we did not include sources describing only specific technical components of an LHS, such as EHRs, software, hardware or interoperability.

We searched OVID Medline®, Embase® and IEEE Xplore® from January 2007 to January 2020. We originally searched databases on March 24, 2019 and updated searches on January 2, 2020. We also reviewed all papers published in the following journals: Leaning Health Systems, eGEMs, and BMJ Quality and Safety. We then reviewed all references from Budrionis and Bellika (2016) and other systematic reviews that were found from our database search [8]. We also used Google Scholar to find articles that cited the IOM’s (2007) LHS report [6]. To capture gray literature we also performed a Google search in August 2019 for ‘learning health systems’. To confirm whether an LHS was eligible for inclusion in our review, we reached out to some authors or website contacts.

We consulted a librarian at Western University on the search strategy (Additional file 2). For Medline® and Embase® we used keywords for different variations of LHSs including ‘health’ versus ‘health care’. We also included Medical Subject Headings (MeSH) terms for ‘learning’ and ‘health care delivery’. We used search terms for primary care in the IEEE Xplore® database search but not for the other databases, since this narrowed the articles retrieved and did not capture integrated health systems. After all eligible data sources were found we completed a targeted search using Pubmed and Google for other sources of information further describing these LHSs. This was to ensure completeness for our data charting. For example, an article found through our search may have discussed the LHS in detail but lacked information on the organization or health system in which it was based.

All articles from the database search were uploaded to Covidence [12]. DMN screened all article titles and abstracts through Covidence and selected ones eligible for full-text review, and then completed the full-text review. ZB further reviewed full-text articles for eligibility, and both authors discussed discrepancies. DMN created the data charting form and tested it on five articles to ensure all information would be captured. ZB charted the data from the eligible articles and websites, and DMN reviewed and confirmed all information entered in the spreadsheet against the original data source. The charted data elements are described in Additional file 3. We summarized data in table format (overview of LHSs, and challenges and solutions) and included a description of each learning health system with examples of primary care specific learning initiatives in the text. In some cases, we reached out to authors or website contacts to further confirm details.

Through our database search we retrieved 1924 unique articles plus an additional 51 from other sources. The number of articles excluded from the database screening, and the reasons for exclusion are shown in Fig. 1. Confirmation of eligibility by reaching out to authors confirmed inclusion for one data source and confirmed exclusion for two. We only searched up to page 8 in Google for our internet search, since the results were not relevant after this.

We identified 21 LHSs including primary care that met our definition, from 40 peer-reviewed research articles and 22 websites (Table 1). Most (18/21) self-identified as an LHS, where three did not: Health Care Systems Research Network (HCSRN), High Value Healthcare Collaborative (HVHC), and Intermountain Healthcare [13‐15].

Translational medicine and patient safety (TRANSFoRm) was the only LHS included in our review that operated only in primary care [61]. The remaining LHSs are vertically integrated health systems or networks that deliver care through hospitals, universities and primary care. Although both are integrated health systems, the papers describing Johns Hopkins’ and University of Wisconsin’s LHSs focus on primary care. Three of the LHSs identified were in Europe [53, 59, 61], where the remaining were in the United States.

All but one of the included LHSs appear to be financially sustainable, with the learning activities from most organizations funded through care delivery, rather than external, project-based or time-limited funding. The exception is TRANSFoRm, which is a time-limited research project dependent on external funding [18, 61]. However, even internally-funded LHSs, such as Geisinger Health System, raised concerns about financial sustainability of their learning activities [75].

We also identified 16 additional data networks / organizations (based on 14 peer-reviewed research articles and 16 websites) that included primary care data, but did not meet our definition of a fully-functioning LHS, since they were mostly platforms for conventional research and knowledge translation, rather than building internal systems for directly identifying gaps and cycling knowledge to improve care (Additional file 4). The majority of these (12/16) were projects funded through the National Patient-Centered Clinical Research Network (PCORnet), with the initial aim to conduct large-scale comparative effectiveness research [76]. We included all 16 in an additional file as “potential LHSs”, rather than excluding them from our review, since many are working towards an LHS by establishing a data sharing process. However, our review and search strategy targeted LHSs, so the identified research / data networks and organizations may not be comprehensive.

We identified 13 integrated LHSs in our review, all of which are in the United States.

Geisinger Health System – a large not-for-profit integrated health system in Central Pennsylvania and Southern New Jersey encompassing eight hospital campuses – was an early adopter of LHS approaches [19]. In Psek (2015), they reported a framework for nine components of an LHS and outlined how Geisinger meets these components including data and analytics, people and partnerships, patients and family engagement, ethics and oversight, evaluation and methodology, funding, organization, prioritization, and deliverables [20]. We did not find any documented primary care-specific LHS initiatives for Geisinger.

Kaiser Permanente is another large not-for-profit integrated health system serving members across eight states and the District of Columbia [77]. We identified many examples of LHS activities among the different Kaiser Permanente sites [29, 32]. Kaiser Permanente Southern California developed an Outpatient Safety Net Program, which uses clinical surveillance software to regularly scan outpatient EHR data to identify care gaps [33]. Similar initiatives included screening for new diagnoses and ensuring follow-up on positive tests, such as cancer screening, kidney disease, and Hepatitis C, and identification of potentially harmful medications and interactions, or those that require laboratory monitoring.

To address an identified problem of missed appointments, Kaiser Permanente Colorado investigators conducted a randomized clinical trial and showed that phone call and text message based reminders to patients significantly reduced missed appointments [27]. They also developed and externally validated a prediction model to identify those most likely to miss their appointment.

The Veterans Health Administration (VHA) is the largest integrated health system in the United States consisting of a national network with more than 1200 centers (including over 1000 outpatient clinics) serving 9 million veterans [43]. The VHA uses a combination of research and quality improvement to improve care [44, 45]. They demonstrated one way to embed research into clinical practice through an initiative called Point of Care Research (POC-R) [46]. This initiative facilitates pragmatic trials using EHR data by fully embedding these trials into practice. For example, when a patient is eligible for one of the ongoing trials, a notification is sent through the EHR asking the clinician if they want to randomize the patient or not. A research coordinator then obtains patient consent and all data is collected through the EHR.

Another VHA initiative called Quality Enhancement Research Initiative (QUERI) is an internally-funded program to help the VHA translate research into practice more quickly and efficiently than traditional knowledge translation [45, 47, 48]. Four priority research projects were: 1) home- and community-based care for Veterans at risk of nursing home placement, 2) risk mitigation for patients receiving opioid prescriptions, 3) targeting care for patients at high risk for suicide, and 4) a telehealth tool to improve access to dermatology services. An Implementation Roadmap was also developed to scale up these initiatives and to help clinicians sustain them in practice [48]. This Roadmap was developed based on a review of existing frameworks and successful implementation strategies and based on the expertise of QUERI programs. However, further research and evaluation is needed to confirm and further refine Roadmap components [48].

Intermountain Healthcare is a not-for-profit integrated health system serving patients primarily in Utah, southern Idaho, and southern Nevada [23]. Intermountain Healthcare was an early adopter of a data warehouse in 1998 to measure best practices and costs, and to inform quality improvement initiatives [15]. They continue to use their data to identify gaps and quality improvement efforts to inform clinical decision making [24]. One example is the development of an Area Deprivation Index (a proxy for socioeconomic status), which was evaluated in a prediction model and found to help identify those who would benefit most from enhanced care management services [78].

At Johns Hopkins there are 240 primary care physicians serving over 250,000 patients across Maryland [26]. McGuire (2019) describes the evolution of developing an LHS within primary care at Johns Hopkins, including clinician experts and a quality analyst who provides EHR support, development of a team-based learning culture among staff and clinicians, improvement of patient experience, and incentives for participation in learning activities among clinicians [26]. We could not identify any documented specific LHS projects for Johns Hopkins.

The University of Wisconsin developed a framework for learning health system development and sustainability through their Health Innovation Program, which focuses on research being an equal component to health care [41]. One example using this Health Innovation Program framework is the development of a health case management program to improve primary care for patients with complex medical or social care considerations. Through this program, 20 nurses and social workers were hired as case managers to provide telephone support to these patients. They also developed a prediction model to identify patients who would benefit most from this program. This health case management program is being scaled up across all their primary care clinics, and future evaluations will assess the impact of this scale up [41].

NYU Langone Health has six hospitals and eight primary care and specialty centers [38]. They implemented a rapid-cycle initiative to evaluate existing practices in randomized trials to determine whether they should continue or be modified, and to eventually test new quality improvement initiatives. Over a one-year period, ten existing delivery practices were evaluated – most within primary care – including prompts for flu vaccines and smoking cessation, mailed reminders for appointments, comparison of different telephone scripts for annual visits, and messages to patients to complete health surveys [39].

The Agency for Healthcare Research and Quality (AHRQ) in the United States highlighted the following three large integrated healthcare organizations on their journeys to becoming LHSs: 1) Baylor Scott and White Health – the largest not-for-profit health system in Texas, 2) Denver Health – providing comprehensive care to a third of all Denver residents, and 3) University of Utah Health – an academic healthcare system in Salt Lake City, Utah [16, 17, 40]. Baylor Scott and White Health is planning to transition all their sites to a single EHR and create value-based dashboards for all primary care physicians to standardize information [16]. Denver Health has launched a Quality Improvement Review Committee to review proposals for new initiatives [17]. Denver Health and University of Utah are both using their EHRs to benchmark progress in comparison to baseline data and other leading institutions [17, 40]. University of Utah Health is also using monitoring systems to provide real-time feedback on system-wide issues [40]. However, all three organizations do not seem to have used internal data systems to evaluate the impact on patient care, outcomes or costs of these changes to their practices.

TRANSFoRm is a network of 21 partner organizations from 10 European countries focusing on primary care research [61]. TRANSFoRm was originally developed for three purposes: 1) prospective study recruitment including randomized trials, 2) retrospective analyses, and 3) decision support for clinical care [62, 63]. Through TRANSFoRm, an electronic solution was developed and validated for standardized and automatic recruitment and data collection for pragmatic clinical trials embedded in EHRs, which is now being deployed in the United Kingdom [79]. The decision support component focused on embedding diagnostic support for primary care clinicians within their EHRs advising on earlier cancer diagnoses [64]. An expansion of TRANSFoRm includes ROAD2H, which is an LHS that will provide decision support combined with local clinical guidelines for primary care to low- and middle-income countries [65].

We identified two other learning health data networks also in Europe: the Swiss Learning Health System (SLHS) in Lucerne, Switzerland and Connected Health Cities in North England, United Kingdom [53, 54, 59]. Projects are being advanced within the SLHS including themes of innovation in service delivery, health promotion and prevention, and health systems guidance and intelligence [60]. The Connected Health Cities was a pilot project leading to over 10 million citizens across North England with connected health records [53]. Sixteen clinical pathway projects have been developed using these health records to help improve patient care. One example is the development of a National Antibiotic Prescribing Dashboard, which uses anonymous data to allow primary care physicians to compare their antibiotic prescribing practices to national and local averages, and allows them to identify higher risk patients [53].

In the United States, we identified two university-based learning health networks that include primary care: Northwestern University Clinical and Translational Sciences Institute (NUCATS) and Health Sciences South Carolina [55, 57]. NUCATS supports clinical trials, community-based research, as well as dissemination and implementation of findings [57]. We could not identify any documented primary care-specific LHS initiatives for NUCATS and Health Sciences South Carolina.

We identified three network of networks, where two of these – the Healthcare Systems Research Network (HCSRN) and the High Value Healthcare Collaborative (HVHC) – each consists of 19 different healthcare systems across the United States, and Optum Labs which standardizes EHR data from across 52 health systems in the United States [13, 14, 72]. Optum Labs describes the ability to use their healthcare data to conduct “N of 1” studies, so healthcare providers can generate evidence that is directly applicable to complex, unique patients at the point of care [72]. We could not identify any documented system level improvement activities nor any primary care-specific LHS initiatives for HCSRN and HVHC.

The LHSs included in this review identified challenges that they experienced or anticipated and potential solutions to these challenges in regards to data standardization or quality, ease of data access and use, financial sustainability, promoting a culture of learning, involvement of patients and the community, consistency across different sites within the same organization, prioritization of learning initiatives, the use of EHRs for quality improvement, and the move toward embedding evaluation and implementation of improved practices into usual care rather than traditional research projects (Table 2).

Large integrated health systems in the United States and research / data networks in the United States and Europe provide some of the leading examples on developing an LHS within primary care. We identified only one LHS that operates exclusively in primary care, which is a research-funded initiative in Europe, TRANSFoRm. It is not clear from the data sources how integrated TRANSFoRm is with healthcare delivery organizations, since their partners are primarily academic organizations [62‐65, 79]. The PCORnet projects demonstrate the United States’ motivation to move towards a national-level LHS, yet many of these networks need to apply more rapid quality improvement initiatives to keep moving in this direction rather than relying on traditional research [76, 81].

We identified challenges to initiating or sustaining an LHS in primary care and some potential solutions to these challenges. However, these solutions were presented at a high level and may need to be described in more detail to be useful for other organizations and networks looking to apply them.

Integrated health systems are at an advantage of having access to large amounts of healthcare data and delivering care to patients, compared to organizations that have access to healthcare data but are not directly responsible for delivering care. Furthermore, learning activities are generally funded by patient care revenue, so savings or improvements can be directly applied, thus providing an ideal platform for a self-sustaining LHS.

On the other hand, data networks develop the mechanisms and find the resources to link and analyze healthcare data, and although the data contributors to the network may be healthcare organizations, the networks themselves do not provide patient care, making it difficult for networks to directly develop interventions to improve care across all member organizations. This was why many of the data networks identified in this review did not meet our definition of a fully-functioning LHS. The data networks that were eligible provided some examples of using data to improve healthcare or at least plans for how they will become a fully-functioning LHS, but they generally showed less maturity than the integrated health systems. Network of networks, such as Optum Labs [72, 73], also use a different approach than the integrated health systems, which has some advantages and disadvantages. They provide a service for data integration and analysis, and a platform for collaboration and research. Although initial investments are required to establish these networks, they could become financially sustainable through the researchers and organizations who pay for their services. Similar to data networks, a disadvantage is their disconnect between healthcare provision and thus their limited ability to directly impact patient care at the system level, as compared to integrated health systems. For both types of networks, the separation between data analysis and responsibility for care improvement may limit their ability to design and evaluate effective interventions to improve care, along the same lines that integrated health systems have achieved.

The purpose of this review was to describe LHSs in primary care, however, we found most of the identified LHSs included other healthcare settings. This could be an advantage, since integrated organizations can apply learnings to the appropriate level more easily than primary care providers who are not closely linked to the rest of the system. Furthermore, many of the learning initiatives described included specific projects and not overall plans for whole system improvement in primary care. Geisinger is one of the longest established self-identified LHSs, and not surprisingly describes how their organization met all the criteria for an LHS, but even here there is little information on how this extends to the primary care setting [20]. Finally, the learning initiatives described in this review mainly include sequential research projects or quality improvement activities, but for an LHS to be as productive as possible it needs to move beyond sequential projects to continual improvement without sacrificing quality. NYU Langone Health seems to be moving in this direction with their rapid-cycle testing of various healthcare interventions using randomized clinical trials [39].

There are some limitations of our review that should be noted. We used keywords for LHSs to identify eligible studies. However, this may have biased our review to identifying more LHSs in the United States where this concept is more established, rather than other parts of the world where organizations may not realize that they are functioning as an LHS. It was beyond the scope of this review to seek out LHSs that were not self-identifying as such, although we did find a few using our search strategy. This presented another challenge of identifying organizations and networks that are true LHSs that go beyond just a data network, as there is not a clear definition of an LHS in the literature that can be easily applied to assess organizations as such. We created a definition based on existing literature and then refined this through the current search as we identified organizations that exemplified this criteria. We recognize that our definition may be a simplification of the concept. For example, Geisinger identified nine criteria that are required for an LHS [20]. However, most of the identified data sources did not provide enough details to assess whether or not they met all nine criteria, so if we applied this definition we would have underestimated the LHSs included in our review. Furthermore, LHSs also exist on a smaller scale, including solo-practicing physicians who use their data to learn and provide better care, but this was beyond the scope of this review.

We restricted our searches to English language only, which may also have biased the review towards English-speaking countries such as the United States and the United Kingdom. Finally, we only had one individual available at the time to perform the title and abstract review, even though guidelines recommend that two people should be involved at this step.