Domains associated with successful quality improvement in healthcare – a nationwide case study

The method of the present study had four steps. First we developed a questionnaire for improvement teams. The instrument was based on a qualitative study of the conditions for change among 19 participants of the learning collaboratives of the Norwegian Medical Association (Sub study I, published in 2011) [23]. Second, we submitted the questionnaire to the 189 improvement teams of the same learning collaboratives. Third, we analyzed the data by comparing the reported conditions for improvement in the organizations of the successful projects versus the other (comparator) projects. We already knew the success level of the 189 projects from the validation of a Change Process and Outcome Scale instrument which was published in 2015 (Sub study II) [32].

Between 1998 and 2011 The Norwegian Medical Association sponsored eight hospital related improvement collaboratives to support quality and safety improvement efforts in clinical environments (Table 1). The improvement collaboratives were based on the Breakthrough Series model of the Institute for Healthcare Improvement, aiming to accelerate improvement beyond what had been achieved by traditional educational approaches [34]. The model has two dimensions: the learning collaborative method (national level), and the systematic approach to continual improvement (organizational level) [11, 22, 35‐42] (described in Additional file 1: Supplement 1). Each collaborative lasted from 6 to 9 months and engaged clinicians from 15 to 30 healthcare organizations who met to learn from each other and from recognized experts in specified topic areas (Table 1). The participating improvement teams sent 2–4 representatives from different disciplines involved in the topic (at least one physician) to three collaborative learning sessions, where the relevant subject matter experts of the collaborative (medicine, nursing, psychology, etc.) demonstrated the quality gaps within the topic to the participants. A team of 10–15 improvement knowledge experts (coaches) guided the improvement teams at, and in between the learning sessions.

The questionnaire was developed to identify the activities and conditions associated with successful quality improvement initiatives, and to study the effectiveness of the continual improvement method (Additional file 1: Supplement 1). The instrument and its development is described in details in Additional file 1: Supplement 2.

The first sub study was published in 2011 [23], and provided us with a large amount of relevant comments, and enabled us to develop a validated questionnaire reflecting the most interesting conditions for change reported by clinicians telling their improvement project stories from their own organizations, after participating in a learning collaborative of the Norwegian medical association.

The questionnaire was submitted to former improvement team leaders between 2 and 4 years after the end of each improvement collaborative. We had access to their e-mail addresses from the improvement collaborative participant list. A link to an on-line questionnaire was e-mailed to the improvement team leaders. They were asked to think back on their improvement project and the promoting and inhibiting conditions for quality improvement that they encountered, and to show their level of agreement with the focus group comments that were included in the questionnaire.

If a Word-version of the questionnaire was preferred, the respondents returned their filled-in questionnaires by e-mail or “surface mail”. In cases of non-response from team leaders, we contacted other participants from the same team. In 36% of the teams, late responses lead to more than one response from the same team. Because the responses from team members mostly reflected different professions, and the inter-rater reliability of the same team ranged from poor to strong, we decided to let each team be represented by the average ratings of its responding members.

This study is neither an experiment, nor a study of the experiments of others, aiming to bring evidence to the success of the projects in our material. This is a study of the conditions for making successful changes. The aim is to learn from healthcare professionals in the improvement teams of those projects who have been able to document improvements.

The second sub study was published in 2015 [32]. Seven improvement experts from different healthcare professions alternated in participating in a four-person review team. The reviewers were two physicians (BN and TSH), three nurses (GSH, ASB, and EA), one psychologist (LdG) and one bioengineer (AS). The improvement experts were not involved in the evaluation of projects they had participated in with coaching or other kinds of support. In this study we explored the final reports of the improvement teams. We developed a checklist to structure the study according to the recommended improvement method (Plan- Do-Study- Act-cycles) [38], to make it easier to discuss our observations an reflect on our different assessments of the project. The criterion to be classified as successful was to document significant improvements by recognized measurement methods, based on a clear linkage between vision, aims, change efforts and measurements.

We found that 72 projects (38%) were successful, ranging from 17 to 60% within each of the eight collaboratives. A majority (78%) presented their outcomes as a shift in the level in the desired direction on a control chart.

We analyzed the association between the assessments of the improvement teams (responses to 25 selected questions) and the success level of their projects. First, a logistic regression analysis was used to analyze the association between success and each of the 25 questions. Second, a factor analysis with Varimax rotation was used to identify the underlying structure of the 25 questionnaire items. Domains were extracted with an Eigen value greater than one. Kendall’s-tau-b correlation revealed that the conditions for a principal component analysis (PCA) were present. Third, when analyzing the multivariate associations between the five domains and success. Logistic regression analyses were performed, with success as the dependent variable, and the success domains as independent variables. Only domains which were significant in a bivariate analysis (defined as p < 0.05) were included in the multivariate regression analyses. The results from the regression analyses are presented as odds ratios with 95% confidence intervals and corresponding p-values. A significance level of 5% was used. All statistical analyses were performed using the software package IBM-SPSS version 21.

First, in a logistic regression analysis of the answers to the 25 questions of the questionnaire (Additional file 1: Supplement 2) we identified the variables which were significant associated with success. Two variables were found in the final model: (Q12) Good guidance & help with measurement, and (Q7) Someone in the improvement team enjoyed working with measurement (Table 3).

Second, to disentangle what combination of variables are underpinning successful improvement efforts, we performed a factor analysis of the 132 responses to the 25 questions. This analysis produced five domains: Domain I: “Measurement and Guidance” (nine variables), Domain II: “Leadership engagement”(five variables), Domain III: “Professional environment” (seven variables), Domain IV: “Group process” (two variables), and Domain V “Leadership impact” (two variables) (Table 4).

Third, we studied the quartiles of the domains in the successful projects and compared the scores from the 54 successful projects with the 78 comparator projects within the five domains. Two domains were significantly associated with success: Domain I Measurement & Guidance (p = 0.002) and Domain III Professional environment (p = 0.002), (Table 5).

Finally, we made a logistic regression analysis of the five success domains. As shown in Table 6, two domains were found in the final model: “Measurement and Guidance” and “Professional environment”, confirming the findings of the crude analyses displayed in Table 5.

The complexity of our findings is displayed in Table 7 presenting the combination of variables that are underpinning the two success domains, illustrated by the proportion of successful and comparator projects scoring on the positive side of the scale (4 + 5) to each variable.

The first success domain “Measurement & Guidance” cover the two success variables from the first regression analysis: Good guidance & help with measurement, and Someone in the improvement team enjoyed working with measurement. In addition the findings suggest it was easier for the successful projects to get hold on their coach when needed (Q10), an availability they assessed as important to succeed (Q11). Further, did the control charts appear to be easy to communicate to their peers in the site (Q20), assessed as important when trying to make successful improvements (Q21).

The second success domain “Professional environment” indicate the importance of presenting patient focused aims when trying to engage of the professional environment in the improvement efforts (Q16), and the importance of presenting measurement to maintain motivation (Q23). Regardless of their success level, 83% reported they had been able to organize their improvement efforts well, in spite of a limited resource situation (Q5) (Table 7).

Our findings reflect the intellectual underpinnings of the continual improvement method presented in Additional file 1: Supplement 1. High performing professional environments were successfully pursuing patient-centered quality improvement if they had access to a combination of: (a) knowledge of best practice, provided by professional subject matter experts, (b) knowledge of current practice provided by simple measurement methods, learned from (c) improvement knowledge experts who provided good guidance and help with measurement, and made the team able to organize the improvement efforts well in spite of the difficult resource situation (time and personnel).

Our study underscores the power of good guidance and help with measurement. In contrast to most learning collaboratives abroad, The Norwegian Medical Association invested in a team of 10–15 improvement knowledge experts (coaches) to guide their improvement teams [23]. The coaching team met regularly for education and training. A system of mentoring was developed to enable experienced coaches to support the novice coaches (Additional file 1: Supplement 4).

The present study highlights the importance of using measurement to understand and reflect on the variations in current practice, and to monitor the target process continually to maintain motivation for change. Learning from the final reports of the improvement projects, the sub study indicate that successful results are connected to a clear linkage between vision, aims and proper measurements, clear and understandable improvement efforts, and the ability to communicated this all to others in an understandable way [32]. Our findings support the findings of others indicating that by measuring and monitoring variation and change with control charts, it is easier to understand and manage performance from week-to-week, communicate progress, and motivate colleagues to sustain the improvements [43‐48].

Our The glue for interprofessional collaboration is a common interest in the patient’s welfare, which has been emphasized as crucial by others [49]. We found that 92% of the 198 responding physicians, nurses, psychologists and other members of the improvement teams found patient-centered targets of “great” or “very great” importance for engaging their colleagues in quality improvement (Q16 Table 7).

We have found that successful quality improvement efforts depend on certain conditions for change in the participating organizations that to a certain degree have been facilitated by the national learning collaborative. However, if continual improvement efforts are to become part of everyone’s work in healthcare, an infrastructure for improvement that at least is providing similar conditions for change in the local context is essential. The infrastructure should include: (a) a system that promotes leadership’s engagement at every stage of the improvement work, (b) provides easy access to clinical data needed for improvement measurement and reflection, and (c) provides qualified improvement guidance to frontline clinical improvement teams [23].