A Canadian evaluation framework for quality improvement in childhood arthritis: key performance indicators of the process of care

Juvenile idiopathic arthritis (JIA) is an inflammatory arthritis that affects approximately 3 in 1000 Canadian children [1]. Timely diagnosis and treatment in a pediatric rheumatology center is key to improving outcomes of patients with JIA. While contemporary treatments have resulted in excellent outcomes for many patients, there can be variability in outcomes with a severe disease course in about 20% of patients in contemporary cohorts [2]. For some with JIA, disease may persist into adulthood [3‐5]. In a Norwegian study, after 30 years of follow-up, persistence of active disease or medication use was seen in 41% of patients with JIA, with up to 28% having a high symptom state [6]. Consequences of inadequately treated disease include pain, functional limitations due to joint deformities or damage, growth abnormalities, and psychological impacts [7, 8]. Beyond the burden on patients and families, JIA can also be a costly disease. Patients who fail first-line disease-modifying anti-rheumatic drugs (DMARDs) are often treated with biologic therapies, which are highly effective in improving outcomes, but are costly [9]. Indeed, tumor necrosis factor inhibitors (anti-TNF), the biologic DMARDs most often used in JIA, rheumatoid arthritis (RA), and inflammatory bowel disease, are the drug class that accounted for the highest proportion of public drug spending in Canada in 2018 [9].

Variability in practice and care can contribute to suboptimal patient outcomes. Over the last number of decades, there has been increasing interest in measuring and monitoring processes and outcomes of care to reduce unwarranted variability and improve quality of care, defined as “the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge” [10]. In the USA, the American College of Rheumatology (ACR) published a White Paper on Quality Measurement in 2011 highlighting the quality landscape for rheumatologists and prioritizing areas for measure development [11]. In that White Paper, JIA ranked 3rd as an area for future ACR quality measure development following RA and osteoporosis. Despite this, there currently exist no ACR-endorsed JIA quality measures [12]. Nevertheless, groups such as the Pediatric Rheumatology Care and Outcomes Improvement Network (PR-COIN) in the USA and the British Society for Pediatric and Adolescent Rheumatology (BSPAR) in the UK have proposed measurement tools for quality improvement in JIA [13, 14]. In Canada, the Arthritis Alliance of Canada (AAC), a non-profit group representing 36 arthritis stakeholders’ groups, developed a set of system-level performance measures for inflammatory arthritis including JIA [15]. The measures focused on early access to care and treatment, and 3 of the measures were applicable to JIA. However, the measures addressed only wait times, yearly follow-up, and pediatric rheumatology access within 1 year [15].

The objective of this study was to review the current landscape of JIA quality measures and through expert panel consensus to define key performance indicators (KPIs) that are important and feasible to collect for routine monitoring of JIA care in Canada. The KPIs will be implemented and tested in future studies and will help assess strategies for care improvement. The KPI framework also represents a first step in the evaluation of individual patient and health economic outcomes.

The evaluation framework was developed over 3 phases using a modification of the RAND Corporation/University of California Los Angeles/University of California Los Angeles (RAND/UCLA) Appropriateness Method [16]. A summary of this process is shown in Fig. 1.

The present study identified 10 KPIs that examine important processes of care for JIA that will be tested in future studies. This work builds upon previous national efforts to develop a measurement framework for monitoring and improving care for patients with inflammatory arthritis that was developed in collaboration with the AAC [15]. The AAC System-Level Performance Measure set included 6 measures that have been used to evaluate early access to care and treatment for inflammatory arthritis, including JIA [24‐28]. All 3 of the measures previously included in the AAC measurement set (wait times for JIA care, percentage of patients seen within 1 year of diagnosis, and percentage of patients seen in yearly follow-up) were retained in the current measurement set. While many of these measures build upon existing published measures, some modifications were made to better reflect Canadian practice and/or current guidelines. Furthermore, a new measure was proposed to ensure measurement of disease activity with the cJADAS to optimize efforts in treating JIA to target [29].

Internationally, this work is complementary to a number of national and international efforts to monitor and improve the quality of care for patients living with JIA. For example, PR-COIN is a quality improvement learning network that uses patient registry data to inform quality improvement strategies to optimize processes of care and patient outcomes and includes 20 centers in the USA and Canada [30]. Beyond measurement of processes and outcomes, PR-COIN uses a number of strategies to improve care including pre-visit planning, strategies and toolkits for self-management, and patient/parent engagement [30]. PR-COIN’s quality measurement framework includes several process and outcome measures adapted from a set of 12 measures assessing processes of arthritis care that were also considered in the present study [20]. They were developed by a working group including representatives from the ACR, American Academy of Pediatrics, American Board of Pediatrics, and the Association of Rheumatology Health Professionals [20]. PR-COIN uses sophisticated strategies for audit and feedback based on their measurement framework and improvement science methods to drive change. Importantly, the network sites have been mapping captured data to electronic records to facilitate data entry and automate reporting.

In 2010, the BSPAR developed standards of care for children and young people with JIA that addressed the empowerment of patients, identification of JIA, referral to pediatric rheumatology care, access to a pediatric rheumatology multidisciplinary team, eye screening, access to treatment including joint injections, regular review to target “tight” control strategies, clinical networks and arrangements for shared care, and the care of adolescent patients [31]. An audit in 10 pediatric rheumatology centers using retrospective chart review demonstrated variable adherence to the standards of care including delays in access to care [32]. The audit highlighted a need for consensus on measurable JIA quality indicators, which prompted the development of the BSPAR National Audit Tool for JIA which was funded by the Health Care Quality Improvement Partnership, an independent organization that aims to promote quality in healthcare [19, 33]. The audit tool contains not only quality measures but also prospectively collected patient and care giver reported outcome measures and experience measures (PROMs and PREMs) in a questionnaire. In parallel, a core dataset for JIA was developed with the vision of collecting standardized items in the audit tool to facilitate quality improvement and research efforts [19]. In the present study, items from the BSPAR audit tool were considered for inclusion in the framework; however, none were included in the final set of KPIs.

While our study represents the first Canadian national effort to define KPIs that are feasible and important to measure to drive improvements in processes of care in JIA, the ultimate goal is to improve patient outcomes. We did not, however, include any outcome KPIs in our framework at the present time for several reasons. Firstly, we relied on published quality measures as a starting point for our framework to ensure measures were comparable with other centers. At the time of our study, outcome measures from PR-COIN were not published and available in the public domain for consideration of inclusion [20]. The outcome measures reported from the BSPAR audit tool relied heavily on patient experience or outcome questionnaires, which were deemed less feasible to collect in our healthcare setting. Secondly, agreeing upon and monitoring of the process measures was considered a first step to the evaluation of outcomes and setting benchmarks. Following our study completion, the American College of Rheumatology/Arthritis Foundation guidelines for treatment of JIA were published [34]. These were unfortunately not available for consideration during our study; however, none of the KPIs currently included in our set addresses treatment at this time and it is unlikely based on the KPIs considered by panelists that the guidelines would have altered the final resulting set. In the future, the updated guidelines should be considered when developing additional JIA treatment KPIs. Lastly, there exist a number of challenges for the development of outcome measures including but not limited to outcome attribution, risk adjustment, and defining the period of risk (reviewed in detail by Suter et al. [35]). We therefore propose that this set represent a starter set for data collection and monitoring as the processes required for documenting important patient outcomes including pain, disease activity, and function are included. Future projects would work to define appropriate outcome measures and benchmarks for care.

It should be noted that while all KPIs considered for inclusion in the framework were rated highly by our panel in terms of importance and priority for measurement, there were significant concerns around the feasibility of measurement of some KPIs, which lead to their exclusion. This should not be interpreted as a lack of importance of these measures necessarily; however, this highlights that improved methods of data collection could result in the addition of some of these measures to the framework at a future date. For example, while critical to high-quality JIA care, uveitis screening documentation was felt to be a challenge in many centers, which could have led to falsely low measure reporting and hence lower feasibility ratings by our panelists. Future work should therefore focus on improving reliable ways to track screening in electronic records. There was also skepticism about how patient experience and quality of life questionnaires could be used to improve patient care and/or health leading to the exclusion of measures relating to these concepts. A lack of standardized questionnaires in these domains was also highlighted, as were concerns about length of existing questionnaires and burden to patient and families receiving care and collection of such questionnaires is not currently routinely completed in practice. Further research in the area of patient experience and quality of life can be used to drive clinical improvements and may help inform future performance measurement in this area.

Our KPI framework was developed building upon existing frameworks in the USA and the UK and highlights important processes of JIA care with measures adapted to the Canadian context. The feasibility of capturing these measures will be examined in available data sources, including the data collected prospectively as part of a national study (UCAN CURE: Precision Decisions in Childhood Arthritis). Ultimately, these measures may also inform routine clinical practice and quality reporting. Importantly, the collection of these process KPIs systematically will allow us to evaluate the impact on patient outcomes and inform the development of meaningful outcome KPIs in the future.