Evaluating the effectiveness of a multifaceted, multilevel continuous quality improvement program in primary health care: developing a realist theory of change

Over the past ten years the Audit and Best Practice for Chronic Disease Project (the ABCD project) has developed and supported quality improvement tools and processes in primary health care centers across Australia, with a focus on centers that serve predominantly Indigenous populations. We have previously reported on enablers and barriers to participation in the project by health centers, with the objective of informing the development of more effective strategies for supporting uptake [1]. In this paper, our focus is on describing how contextual and other factors may interact to influence service delivery outcomes, particularly the desired outcome shared by most continuous quality improvement (CQI) initiatives, that of improved delivery of recommended care processes.

Variation in the effectiveness of CQI interventions has been commonly reported [2], and a range of theories, models, and empirical studies have advanced the understanding of this variation. However, there is a lack of understanding about how the various drivers of CQI effectiveness that have been identified in the health context interact with one another and with contextual factors to achieve desired outcomes [3]. There are few published empirical studies in this area from primary health care settings, and little clear guidance to policy makers or planners regarding how and in which circumstances CQI interventions could be modified to strengthen desired impacts in primary health care [3, 4]. Our study makes an effort to address this gap.

The ABCD Project was designed to support best practice in prevention and management of chronic disease in Indigenous primary health care services in Australia. There are many different CQI implementation models described in the literature, with the ABCD CQI project sharing characteristics with what has been termed ‘integrated CQI’, that is, CQI models that are multi-site and multi-faceted and aim to achieve change at various levels of the system [5, 6]. Between 2002 and 2006, the project used participatory action research methods at the organizational level to introduce and examine the impact of CQI in 12 remote Indigenous primary health care centers [7‐9]. The extension phase of the project (2005 to 2009) was designed to examine the operational and policy requirements of expanding the ABCD model to other geographic locations, and to other core components of primary health care, including maternal and child health and mental health [10]. The project design drew on diffusion of innovation theory [11] (the ‘how’) and on CQI theory and methods (the ‘what’). By the end of 2009, the tools and processes developed through the project had been used in about 130 Indigenous primary health care centers across the country, including 69 services which were formally enrolled in the project research activities. Ethics approval was obtained from research ethics committees in each jurisdiction.

At the health center level, health centers were supported to conduct annual quality improvement cycles (plan-do-study-act) and a web-based information system provided participants with real-time analysis of their performance data, and capacity to compare it with others in their region and across the project. Health center staff were supported through their annual CQI cycles by state/territory co-ordinators (referred to as ‘hub co-ordinators’) appointed for this purpose. At the regional level, hub co-ordinators based in each of the five regions where the project was operating, had links to regional level management and academic support. In some regions, health boards, or umbrella organizations managing a group of health centers, took on implementation co-ordination roles, including sharing of lessons between organizations within their group. At the national level, the project core, based in an academic institution, undertook ongoing refinements to project design to ensure consistency with national and international evidence-based guidelines, and hosted annual feedback and planning meetings to which all participating health centers were invited. A number of project staff and affiliated researchers were engaged with the development of national and jurisdictional chronic disease policy [12]. Previous project publications have described how the diversity of health centers participating in the ABCD project took up the project in different ways [1], and the considerable variation in their delivery of services that are scheduled at specific intervals according to best practice guidelines (‘guideline-scheduled services’) [7‐9].

In this paper, we combine inductive analytic approaches with principles of realist evaluation [13] to identify fine-grained hypotheses about how, and in what contexts, the ABCD CQI project may have achieved changes in delivery of guideline-scheduled services, focusing on services for diabetes and preventive care. The overall aim of the study is to identify how, why, and in what contexts CQI in primary health care may achieve various outcomes—and to share findings of relevance to implementation researchers, managers, and planners.

Data presented in Figure 2 demonstrate wide variation in baseline level of performance and divergent patterns of change achieved by health centers for overall diabetes and preventive care. As will be evident in the presentation of results below, the participating health centers also showed considerable heterogeneity in their contexts, activities, and outcomes.

We identified three main mechanisms, and seven potential CMO configurations as candidates to explain different patterns of change. Table 1 shows these CMO configurations, together with exemplar quotes, and a summary of the specific inputs of the ABCD CQI project relevant to each of the proposed mechanisms. In the text below, we describe each of the mechanisms; what contexts appeared to trigger or inhibit each mechanism; and what outcome patterns resulted. Specific health center examples are presented in order to illustrate our reasoning. The numbering of the trend lines shown in Figure 2 allows linking of the textual explanation of factors underlying various patterns of change to the trend lines for specific health centers. In the presentation of results below, where contexts are discussed they are numbered in order of citation and referred to as C1, C2.... A similar approach was followed in discussion of mechanisms (M1, M2…). The CMO configurations are reframed into proposition statements, encapsulating the most important findings of use to researchers, managers, and practitioners (see section "List of CMO configurations reframed as proposition statements"). Potential strategies for strengthening wide scale CQI projects in different primary health care contexts and the lines of reasoning that led to them are provided in Table 2.

Using interview data about how key informants interpreted outcome patterns in health centers implementing a multi-faceted CQI project, our study, focusing on ‘middle-range theories’, aimed to articulate more precisely the potential causal linkages between program activities of the ABCD CQI project and outcomes that were achieved. To our knowledge, this is the first time mechanism-based explanation has been empirically developed from a large-scale CQI project in primary health care. While the insights from the paper may be of interest to leaders of smaller scale CQI projects, our key messages are for wide-scale projects and are particularly relevant to understanding ‘integrated CQI’ models in primary health care settings [5].

The ABCD CQI project inputs, and the mechanisms and contexts described are broadly similar to ‘necessary but not sufficient’ common conditions of effective QI reported previously [6]. Our analysis extends this and other broad understandings of the ‘pre-conditions for success’ to suggest how and why different conditions may achieve changes in delivery of care processes through both the design aspects of CQI (project inputs), and implementation context.

A particular strength of our study is its reflection on a key pillar of CQI—that of the use of data as a tool to achieve improvements in quality of care over a number of years of implementation, and how patterns of change are explained by those responsible for supporting CQI implementation. We recognize the possibility of confirmation bias as a threat to the validity of theory-informed data collection, and in considering its impact we note the wide range of literature that informed the evaluative fieldwork and the ongoing emphasis placed on reporting what was actually observed in practice; these aspects of the project would have minimized the potential effect of this type of bias. A related limitation of our study is that our data were derived from a relatively small number of key informants (n = 12) and, although these informants had detailed knowledge of the health centers included in the study and the process of project implementation, they were not necessarily all familiar with the range of effects operating at different levels of the health system. This may have influenced the nature of the data and the level of enquiry that was possible. For example, although the study findings suggested significant effects of social structures and environments in explaining outcome patterns, we were not able to identify the specific situational mechanisms that could explain how social structures and environments may have shaped improvement leadership and management commitment to CQI—or influenced other aspects of context or mechanisms.

Future directions for research suggested by this study include: further empirical inductive studies along the lines of the approach reported here but specifically designed to include a wider range of key informants and able to consider the whole chain of situation, action-formation, and transformational mechanisms [21] appropriate to multi-level multi-faceted CQI interventions; and more focused evaluation studies of CQI that include testing of fine-grained realist hypotheses.

The recommended strategies for modifying CQI projects for greater effectiveness in different contexts flowing from this study include those that should (if our hypotheses are correct) make contexts more favorable to implementation of CQI, or strengthen the action of the key mechanisms that we identified as explaining the patterns of change in our data. These strategies are consistent with previous literature. What our study adds, is that it identifies the context in which strategies should be applied, and this is based on a clear line of reasoning linking identified mechanisms, with program activities, context, and observed outcomes.