Strategies for working across Canadian practice-based research and learning networks (PBRLNs) in primary care: focus on frailty

We carried out a participatory based descriptive study with the Canadian Primary Care Sentinel Surveillance Network (CPCSSN). CPCSSN (www.​cpcssn.​ca) is a pan-Canadian network made up of practice-based research and learning networks [26]. Most regional networks are provincial with the exception of Ontario which consists of six regional networks and Alberta which has two networks. Across CPCSSN there are > 1500 participating family physicians, nurse practitioners and their staff and > 2 million patients who participate.

The CPCSSN has successfully built trusting relationships in its PBRLNs between primary care clinicians and researchers over the past 12 years. It extracts EMR data from consenting clinicians, standardizes the data into a common schema based on internationally accepted ontologies and terminologies and makes the data available for purposes of quality improvement, communicable and non-communicable disease surveillance and research [27]. Participating clinicians are provided reports on their data that compares them to their participating site colleagues and the rest of their PBRLN through CPCSSN’s data presentation tool [28]. CPCSSN has also developed processes that allow participating clinics to securely re-identify and view their own data to enable them to prepare customized lists of patients in specific risk populations [29].

For the purposes of this work, we were interested in creating a reference dataset from which we tested supervised machine learning techniques to derive and validate a case definition on frailty [25]. Supervised machine learning is a type of machine learning that learns by example, where algorithms ‘learn’ how to classify or predict new observations based on previously seen data. The quality of the data used for training machine learning algorithms directly relates to the quality of the algorithms that can be produced and are sometimes referred to as the reference standard. In medical research, supervised machine learning is often used to classify whether a patient has a certain disease based on patient history and past medical charts. The reference standard in this case would contain accurate frailty diagnoses, paired with all known patient information extracted from EMRs. The comprehensive methods and list of algorithms used for machine learning analysis are reported elsewhere [25], but include classification and regression tree, logistic regression, support vector machines, gradient boosted machines, and neural networks.

Previous work using supervised machine learning (classification and regression tree) for the identification of frailty in EMR data by Williamson et al. [23] used data from Alberta, Canada. Tarekegn et al. [30] used Italian administrative health records to develop predictive models for frailty conditions (mortality, urgent hospitalization, disability, fracture, and emergency admission) using neural networks, genetic programming, support vector machines, random forest, logistic regression, and decision tree. The authors found that neural networks showed high performance in predicting mortality and that support vector machines showed high performance in predicting urgent hospitalization. In our study, gradient boosted machines achieved the highest sensitivity (78.14%) and specificity (74.41%) in predicting frailty. Other research has been completed using machine learning methods to classify frailty, but frailty was defined using other instruments such as the Clinical Frailty Scale [23], the Frailty Phenotype [31], or the electronic frailty index [32]. Our study is the first to use pan-Canadian primary care EMR data to create a frailty case definition using machine learning.

In our study, five CPCSSN networks participated in data collection and building the reference set of frail patients: British Columbia (BC), Alberta (AB) (n = 1 network, SAPCReN in Calgary), Manitoba (MB), Ontario (ON) (n = 1 network, MUSIC in Hamilton), and Nova Scotia (NS). Quebec, New Brunswick, and Newfoundland/Labrador were not included within the scope of this study due to funding. Future studies would benefit from including these provinces. Saskatchewan, the Northwest Territories, and the Yukon do not have regional CPCSSN networks at this time and thus were not included in this study. The methods used to create the reference dataset are described here.

The determination of frailty in our study is based on the Rockwood Clinical Frailty Scale (CFS). Primary care clinicians used the Rockwood CFS (Appendix 1) to classify the current degree of frailty observed in a randomly selected subset of their patients. While there are more comprehensive methods to assess frailty, such as the Comprehensive Geriatric Assessment (CGA) scale, they typically take up to  one hour or more to complete. The CGA was discussed as the gold standard to identify patients with frailty, but it was unrealistic to conduct this assessment on a large number of patients due to the extensive time required.

The CFS is a one-page document, taking about 30 seconds to one minute per patient for the clinician to complete. It is simple, efficient, and easy to use in addition to being be a valid and clinically important tool [33, 34]. Patients also do not need to come into clinics to be assessed and there is no associated cost to using the tool. The Rockwood CFS is one of the most widely accepted tools used to identify frailty in Canada. It uses clinical judgement to assign a score from 1 (very fit) to 9 (terminally ill) [33, 34]. The CFS has demonstrated high interrater reliability between physicians and multidisciplinary teams (ICC = 0.97, p <  0.001). It also demonstrates convergent validity in being highly correlated (r = 0.80) with other established frailty tools [33, 34] and has demonstrated predictive validity for mortality rates and entry into institutions [33].

After obtaining consent, clinicians provided assessments of frailty using the Rockwood CFS on a randomly selected subset of their patients who were over the age of 65 and who had had a clinic visit within the last 24 months. Our goal was to collect a reference dataset of 900 CFS scores per network (n = 4500 total) in order to carry out our supervised machine learning analyses, which are reported elsewhere [25].

Each participant completed between 15 and 180 patient assessments using the CFS, depending on how many clinicians were recruited in each network. Lists of eligible patients were either generated by research team members through an EMR search query of eligible patients or clinicians performed their own search query. Clinicians were provided with a list (either a complete list of their patient panel, or a subset) of their patients who met the eligibility criteria. They either completed the assessments on their own, scheduled time with a research team member (author MT), or had a PBRLN working party (MUSIC). Once clinicians completed their assessments, patient IDs were matched to CPCSSN IDs. CPCSSN IDs and their associated CFS scores were recorded on spreadsheets, ensuring no personal patient information was ever taken out of the clinics.

EMR data for all of the patients with a CFS score was compiled into one dataset. All procedures and analyses were approved by regional network directors’ ethics boards, including: University of British Columbia, University of Calgary, University of Manitoba, McMaster University, and Dalhousie University.

Data for this work were also derived from qualitative sources including study team meeting minutes and email exchanges with the study team members. All notes were electronically documented.

Descriptive statistics were used to analyze CFS scores and associated frailty risk factors and demographics. We did not investigate the relationship between variables and the presence of frailty in this work. Future work would benefit from such multivariable analyses. Certain associations between variables of importance and frailty are reported elsewhere [25].

We used the Consolidated Framework for Implementation Research (CFIR) to guide analyses of the qualitative data since conducting a widespread effort to gather CFS is much like implementing an intervention. The CFIR (Appendix 2) is a conceptual framework developed to guide assessment of intervention implementation in a systematic way; it helps to identify factors that might potentially influence intervention implementation and effectiveness [35, 36]. The CFIR is composed of five domains including: 1) intervention characteristics, or features of the intervention; 2) inner setting, or features of the implementing organization; 3) outer setting, or features of the external environment; 4) characteristics of individuals involved; and, 5) implementation process, including strategies that might influence implementation [36]. Thematic analysis of meeting minutes, notes, and conversations with co-investigators of each network reflecting various aspects of the research process was informed by the CFIR.

Table 1 displays results related to the recruitment of FPs. One hundred nine FPs participated in collecting CFS scores across the five provinces. Most of the FPs were recruited through existing relationships with CPCSSN (n = 100), while the remaining were recruited through snowball sampling with support from initial participants (n = 9). There were multiple stakeholders involved in the recruitment process including: CPCSSN regional network directors for each province, research assistants and coordinators, medical office assistants, data managers, and existing physician participants who supported snowball sampling. In Alberta, the head of the Department of Family Medicine committed to involvement for his entire department and provided in-kind support staff to facilitate collection of the CFS scores.

Table 2 displays strategies used to collect data in the 5 provinces. In working with the clinicians who provided the CFS scores, there were several strategies used by stakeholders in each network to ensure timely and accurate assessments of frailty. Common strategies included: providing FPs with lists of the patients they were to assess for convenience; meeting with FPs in person or via email and/or phone bi-weekly or monthly; building on existing relationships with FPs; and attending FP meetings to explain expectations. Key stakeholders involved in facilitating data collection included: study co-investigators from each province, research assistants and coordinators, medical office assistants, data managers, and physician participants who supported other physicians within their clinics. In Alberta, having the Department Head support the project at the departmental level meant that recruitment was trivial, and collection was enabled by using an existing clinical support staff member who was able to remove nearly all barriers to data collection.

Table 3 displays demographic data for patients assigned a CFS score separated by their frail or not frail status. There was a total of 5466 CFS scores provided by clinicians across the five CPCSSN PBRLNs. About 18% of those assessed were identified as frail with a CFS score between 5 and 9. Percentages of frail (11-17%) and not frail patients (82-91%) were similar in all networks, with the exception of the MUSIC PBRLN in ON, who had a higher number of frail patients (25%) relative to the other networks. This finding is likely related to approximately half of clinicians providing care to deprived geographical regions in ON. These percentages of frailty are similar to previously reported frailty prevalence levels [23, 37‐39]. The mean age was 81 for frail patients and 74 for those who were not frail. The majority of those identified as frail were female (65%) and had a significantly higher prevalence of hypertension, dementia, and depression than those who were not frail. Frail patients also had a greater number of medications prescribed and number of healthcare counters relative to non-frail patients. These findings align with other research examining frailty risk factors and outcomes [38‐40].

Table 4 displays demographic data for patients assigned a CFS score separated by frailty severity. When the 5466 patients with CFS scores were categorized according to severity of frailty, 17% were considered vulnerable (CFS score of 1-3), 15% were mildly to moderately frail (CFS score 50-6), and 3% were severely frail (CFS score 7-9). Vulnerable and mild-moderately frail patients are often considered the ideal patients that will benefit from intervention [41]. The constructs of older age, dementia, depression, and a greater number of prescribed medications represented a greater likelihood of frailty presence, as rated by FPs. These findings align with previously reported findings regarding frailty risk factors [38‐40]. There was no obvious pattern between being diagnosed with hypertension, diabetes, osteoarthritis, epilepsy, increasing BMI, or number of healthcare encounters in relation to increasing levels of frailty.

Table 5 displays demographic data for patients assigned a CFS score separated by province. There were 1006 patients (18%) who were identified as frail across the 5 provinces. There was a statistically significant difference (p < 0.05) between the 5 provinces for all frailty risk factors with the exception of diagnoses of hypertension and dementia. The average age of frail individuals ranged from 79 to 84; the average percentage of frail patients who are female ranged from 56 to 71%. These findings align with previous research indicating that frailty risk increases with age and is more likely to occur in females [39, 42].

Table 6 displays numbers of several key stakeholders involved in the research process, including researchers and co-investigators, FPs, data managers, research assistants/coordinators/students, and clinic staff. Numbers of stakeholders varied considerably across the provinces from 2 to 11, nonexclusive of clinician participants.

Some FPs received a random sample of their patients (AB, NS) and provided a CFS score for each patient on their list. Others received a complete list of their patient population who met the eligibility criteria (BC, ON, MB) and provided scores for those patients they were most familiar with and felt most confident assigning accurate scores for. In all networks, FPs had the option of using their EMR records to gather information to assign a CFS score; however, most were comfortable using recall as they were quite familiar with their patients. All networks reported that the FPs were familiar with the CFS tool, but support was available if they had questions.

To facilitate data collection, common strategies were used across the networks, including in-person, telephone, and email communication. In BC, a research assistant was in constant communication with FPs; meetings were scheduled during two of the FPs’ lunch hours and sometimes outside of work hours in which the research assistant supported the recording of CFS scores. In AB, a research team member who was also clinic staff personnel facilitated data collection for each of the FPs. In all networks, research team members (coordinators, administrative staff, students) maintained contact with the physicians and clinic staff to ensure that data collection was complete.

This work did not directly involve patients. The work was completed with individual FPs in primary healthcare clinics and thus there was no networking with external organizations by the FPs. Future utility to patient and primary care was part of the initial engagement discussion with FPs and key to participation.

Networks reported feeling frustration with delays in procedures such as ethics approval as the main barrier to data collection. NS and ON faced several delays with ethics resulting in competing pressure for participating FPs’ attention at meetings where the CFS data collection was to be initiated. Specifically, in ON these meetings (Ontario Health Teams Initiative from the ON Ministry of Health) had major implications for primary care policy and generated intense activity at the time of planned data collection. A decision was made to delay initiation until there was time without pressure from this, and while this resulted in delay in collecting data (about which the network felt pressure), this paid off in subsequent high and enthusiastic uptake by FPs. Furthermore, the ON network was added much later to the project than other networks and thus had to catch up with study planning and protocol. These delays resulted in the research team as a whole unable to adhere to the funders’ deadlines, which had a negative effect on funding decisions.

Additionally, no incentives were provided to physicians in any of the networks with the exception of MB, in which monetary incentives were provided after completion of assessments ($2.50 for each assessment score). For the most part, FPs were motivated by the potential value of this work to primary care.

Although FPs saw the relevance and importance of the work and were interested, it was challenging for them to prioritize engagement and commitment to implementation without additional support. The time burden for FPs of day-to-day patient care is already high. And although the CFS is a low budget tool requiring minimal time, this task was one of many for FPs amidst primary care transformations and immediate clinical tasks. FPs have limited time to devote to clinical tasks and even less time for data recording tasks that do not relate to clinical workflow. It is estimated that, on average, a FP spends 1-2 h of after-clinic time each day attending to administrative tasks [43, 44]. Immediate tasks that impact patient care and workflow supporting this are necessarily prioritised over work that is not remunerated.

Engagement within CPCSSN PBRLNs was generally not an issue and physician-to-physician sharing about the ease of the work led to additional FPs willing to participate. There was potential for high extrinsic incentives related to this work for all networks such as increased stature, goal sharing, changing practice related to frailty identification and management, and potential for publications and knowledge dissemination activities. FPs also saw the value of other quality improvement initiatives they might want to initiate, which was key to engagement.

CPCSSN PBRLN directors facilitated this work within each of their networks; however, students, research coordinators, and administrative staff were key to this process. All research team members were invested in the work and maintained consistent communication with the team through regular team meetings and email communication, as well as with clinics and FPs. The research team took steps to create a climate of trust and respect for FPs’ involvement in the study, while building and maintaining relationships with clinic staff. Communication lines were always left open for feedback and questions throughout the study, and FPs and clinic staff had the contact information of researchers available to them at all times. Research assistants and network leaders also provided multiple reminders and check-ins with FPs either in person or via email in order to ensure timely data collection. In BC, a research assistant met with FPs in person to answer questions and to assist with data recording. In ON, the network leader attended FPs usual practice meetings to explain the study and study processes.

The general plan for this work was created in collaboration between research team members. The implementation of the data collection process and using the CFS was created within each CPCSSN PBRLN. Relationships were key in all networks, leading to snowball sampling and some physicians assisting and motivating others within their clinics (ON, BC). Understanding and discussing the relevance of the project and frailty in a broader sense was key in maintaining these relationships. In BC, one physician was considered a champion. He dedicated additional time and effort to help with this work, including suggesting other FPs who would be willing to be involved, and participating in webinars.

Regular team meetings involving research team members were essential to move the project along and to ensure alignment of procedures. The amount of time taken to collect data and then extract EMR data to be compiled into a pan-Canadian dataset varied considerably by network, ranging from 6 weeks to 10 months from the time of ethical approval. The final dataset was compiled in January 2020.

Adhering to timelines was a key challenge that ultimately affected funding decisions by the CFN. Networks worked off of one another’s ethics applications. Each network individually applied for ethics approval for the work, which created a challenge for the overall study timeline. Ethics approval timelines were not aligned and thus data collection was extremely delayed in both NS and ON (ON faced the additional challenge of joining the research team later relative to other network), resulting in a subsequent delay in national data compilation and analysis. NS was also required to re-apply for ethics due to changes in physicians’ EMR systems. Ethical oversight related challenges are reported as barriers to PBRLN work [11]; we suggest mapping ethics board timelines and having standard block text for components of ethics applications to ensure all team members are clear on the detail and processes of the study. A key lesson learned from these challenges is the need for a protocol for all team members to follow that allows the work to be completed but also allows for a flexible process. Each network was required to tailor the data collection protocol to fit with their FPs’ worldviews. Dania et al. [11] report a significant barrier to PBRLN work as being lengthened timelines. Ethical approval across networks is time-consuming. We suggest a six-month lead time for networks to get through ethics and work out study details and protocols.

Overall, team meetings were successful and disagreements across provincial stakeholders were settled through discussion and consensus. However, collaboration between CPCSSN network leaders, co-investigators, and other research team members was at times challenging. Campbell-Voytal et al.’s [10] study involving seven PBRLNs working in collaborative teams to articulate procedures for PBRN Research Good Practices reports the importance of getting through rocky periods with trust, communication, and handling conflict as key factors to successful PBRLN collaboration. Part of multi-jurisdictional work involves working with differing time zones, competing demands of all those involved, and the increased complexity of research coordination [11]. Efforts were made to ensure that as many team members as possible could attend these meetings; however, this was not always possible. And although the project coordinator recorded meeting minutes and distributed them to the team, there was a risk of team members failing to review these documents, potentially leading to missed information. Regular communication between team members is a key factor for successful research [46, 47] but has been reported as a barrier to PBRLN work due to increased efforts. Dania et al. [11] report a risk for communication and understanding gaps in large networks during research activities, further exacerbated by distance communication. Thus, a key lesson learned for working across networks was to make all possible efforts to maintain communication through the prioritization of regular team meetings with all team members to ensure everyone is aligned in study procedures and timelines.

PBRLNs contribute to knowledge production and utilization and serve as learning communities and drivers of quality improvement [48]. Data on the processes of primary health care are essential to inform decision making at the practice and system level to enable primary health care research [48].

PBRLNs aim to stimulate the development of research that reflects the challenges and context of primary health care practice [48]. Dania et al. [11] report common barriers to engagement in research being a lack of time, low interest in research topics, limited research skills, and competing priorities. In our study, there were differences related to FP’s level of engagement with the work and the assessments they provided. There was no direct incentive back to individuals, which may have resulted in less willingness to participate. Due to competing demands, FPs generally had a low level of motivation to participate, perhaps influencing the accuracy of their assessments. Fagnan et al.’s [49] qualitative study reports that motivation for participation in practice-based research includes personal satisfaction, improving local clinic-based care, and contributing to community and system level improvements. Additionally, Dania et al. [11] report that research engagement is enhanced when practitioners are involved in initiating, designing, and driving research. It is possible that engaging FPs earlier in the research process and involving them in team meetings might have helped facilitate a greater interest.

Conducting studies across networks requires leadership from network directors and research team members. Relationships with FPs proved to be a key factor in navigating the study setting and obtaining timely data, including continuous follow-up and checking in with progress. Clear communication with staff in each network was key, as well as 1:1 follow-up emails to drive the process forward. Early engagement between research team members and stakeholders and meaningful relationships are key to promoting active involvement in and successful uptake of context specific research [47, 50‐52]. Two key lessons learned from these challenges are the need for a laddering of incentives [11] to get through data collection and having a focused timeline across networks to get the job done.

FPs may have also differed in whether they used only recall to provide their assessments, or whether they accessed additional records, thus influencing accuracy of assessment scores. Additionally, some FPs were provided with a complete list of their patients, which was sometimes greater than 500. Although FPs were told they did not need to score all of these patients, but could rather choose those who they felt most confident providing accurate scores for, this process was likely more overwhelming compared to receiving a list of 15 patients to assess (as in AB). Thus, there was a higher intensity of workload required for FPs who needed to provide a greater number of CFS scores vs. a lower intensity for FPs required to provide fewer CFS scores. Recruiting a greater number of FPs in some networks to more evenly distribute patients may have helped reduce this burden. Another notable consideration is the differing numbers of stakeholders involved in each network, ranging from 2 to 11, potentially influencing the burden of tasks within each network and ability to adhere to study timelines.

In AB, the process of CFS implementation was supported by structure. The project aligned with the University of Calgary’s Department of Family Medicine’s goals and values and thus they were able to provide additional support. Alignment of values between stakeholders and researchers in terms of a common purpose and goal is essential in increasing the relevance, meaning, and uptake of the research [53]. The department provided temporary staff to facilitate data collection, data extraction, and data linking. FPs in the department were motivated to complete data collection because this research became a part of their everyday work and they had developed good relationships with team members. Administrative staff from the department were also associated with the clinics and were able to easily gain entry to provide continuous reminders to facilitate data collection. Campbell-Voytal et al. [10] report a key lesson learned in their work with PBRLNs; they found participants involved in a study relevant to their needs and interests will engage and persist despite multiple barriers. Dania et al. [11] also report motivation to participate in research being associated with the presence of a research culture. Similar support and motivation that was evident in AB in other provinces may have made the process of CFS implementation smoother.

One of the key challenges of working across networks is the difference in EMR systems and data that can be used for analysis. Research data quality pertaining to recording, documentation, and coding, and a lack of uniform terminology are reported batters to PBRLN work [11].

Interoperability of EMR systems is also a known challenge to conducting research in different contexts [54‐57]. Often, specific data needs to be excluded when there is a lack of consistency across the various networks. Thus, a key difference among networks that may have potentially influenced study analysis and results was related to using only common data. This study relied on physician documentation in the EMR. Processing algorithms and unique EMR characteristics may have prevented retrieval of some patient characteristics. For example, if assessments related to a functional test for frailty were present within one network’s EMR system, but not in other networks, this data could not be used for national analysis. Free-text data was also not available from all provinces, thus free-text data was not able to be incorporated; the same applied with a large number of blood-work related test results (only hemoglobin A1C, creatinine and estimated glomerular filtration rate was able to be included). In Manitoba, depression and number of unique medications may have been entered into the EMRs differently by physicians, reflecting the differences observed in the number of unique medications prescribed and proportion of older adults with depression relative to the other CPCSSN sites.