A Training Model for Implementing Hepatitis Prevention Services in Substance Use Disorder Clinics: A Qualitative Evaluation

Both training and wait-list teams were sent a Needs Assessment Questionnaire 6 weeks prior to the 1.5-day training (see Online Appendix 3). The questionnaire was designed to evaluate current clinic practices regarding the five implementation goals listed in Table 1. As shown in Table 1, the teams had one or more goals in place at baseline. However, most teams had several, and all had at least one implementation goal that they could work on. Needs Assessment information was utilized during the Action Planning phase of the 1.5-day training to assist the training teams in developing concrete implementation goals that would address gaps between baseline and recommended practice. Teams were strongly encouraged, but not required, to address all unmet goals in their Action Plan.

The first day of the training consisted of didactic presentations and discussion periods covering the risks for liver disease among patients with SUDs, the goals of the LHI, and implementation strategies. The second day was devoted to assisting teams in developing site-specific action plans. Each team was presented with an Action Plan form with implementation goals and Needs Assessment information regarding the baseline practice that they had recorded. Teams were encouraged to list an improvement goal and specific actions that they could commit to within the next month for each row in which the baseline practice did not match the implementation goal.

As part of the training intervention, one representative from each training team was contacted by phone 1, 3, and 6 months after the training. The representative was required to be an SUD clinic staff member with close knowledge of daily processes within the clinic, and was generally the SUD clinic coordinator or an SUD nurse. The purpose of these coaching calls was to assess progress on Improvement Goals and Action Steps recorded in the Action Plan and to coach the team in implementing their plans.

The Minneapolis VA HCRC training program was originally developed for hepatitis treatment clinicians to promote evidence-based evaluation and treatment of patients with hepatitis C. A previous evaluation demonstrated the effectiveness of the program in improving clinic processes and promoting organizational change.22 For the LHI, the original structure of the training program was retained, while the specific content was revised to address a different population and a different goal. Results indicate that the effectiveness of the training model is generalizable to this new population and goal, and that the training program is superior to receiving printed training materials. The results also reveal substantial variability in implementation success among the training teams.

As the number of changes implemented increased, the number of strategies and facilitators reported also increased. While an inverse relationship with barriers might be expected (e.g., more barriers reported, fewer changes implemented), there was no consistent relationship.

Looking at facilitators, it is notable that the teams that were more successful at implementing practice change were more likely to endorse support from the broader system beyond the SUD clinic (e.g., receiving additional staff to assist with implementation, receiving help from other services when requested, and feeling supported by the hospital administration). This suggests that the effectiveness of the training program could be further improved if efforts were made to solicit support from medical center leadership.

These findings provide preliminary support for the value of specific components of the training program aimed at developing a functional team, educating teams in specific active change strategies, and walking them through several steps of the implementation process.

Limitations in the methods of data collection affect the generalizability of these results. Implementation success was reported by one representative from each team, who clearly knew whether their team had been selected to attend the training or wait-listed. This awareness may have created demand characteristics encouraging the training team representatives to over-report their success. Credibility of the reports could have been improved by gathering an independent report from another team member or from an SUD staff member who was not part of the training team. Also, while implementation of a new service had to meet the minimum standards listed in Table 2 to receive “credit”, it is possible that some teams may have implemented higher-quality services than others. Perhaps prioritizing implementation of one or two high-quality services would have greater impact on patient outcomes than focusing on the entire package of services. Since patient-level outcomes were not collected, this remains unknown. In addition, while the training teams were randomly selected from the applicants, wait-listed teams self-selected to complete the project evaluations. Although five of six agreed to complete the evaluations, it is not clear how those that agreed may have differed from the program that declined. Finally, participating clinics were embedded in large VA medical facilities. Generalizability to community SUD clinics would require further evaluation.

In the future, it will be necessary to assess the impact of practice change prompted by this training model on patient-level outcomes such as screening and vaccination rates. To date, all evaluation of the model has focused on practice change, which is assumed to translate into improved patient-level outcomes. It will also be necessary to evaluate the effectiveness of this training model in areas distinct from hepatitis care in order to establish the model as a generalizable method to improve the quality of medical care.

By Bruce E. Landon, MD, MBA, MSc

This paper describes and evaluates a quality improvement program that takes the form of a discrete educational intervention (the Liver Health Initiative) aimed at improving screening and treatment for hepatitis for VA patients enrolled in substance use disorders (SUD) clinics. To understand its effectiveness, the authors evaluated the first cohort of SUD centers that completed the training module.

In this commentary, I outline the importance of including both “process” and “outcomes” components in evaluations of QI interventions, with a particular focus on understanding the importance of knowing both whether the intervention was successful overall and how the innovation was implemented at various clinics.24 In addition, I comment on other design aspects of the intervention that contribute to the usefulness of the evaluation, including the use of mixed methods and the reliability and validity of relying on informant interviews to understand whether an intervention was effective.

This largely qualitative evaluation consisted primarily of telephone interviews conducted with a clinic representative at 1, 3, and 6 months asking about post-implementation success as well as facilitators and barriers. Although used in the evaluation, the calls also were part of the intervention in that they were used as opportunities to coach the teams in implementation of their plans. Teams were categorized according to the extent to which they added services, with 9 of 11 intervention teams rated as achieving high to moderate implementation success.

The first important step to understanding the impact of any QI intervention is to evaluate the impact of the intervention on patient care and outcomes. Even if the educational intervention was reported as effective by participants, its ultimate success as a quality improvement intervention will be defined by the extent to which patients at the enrolled clinics received the recommended screening tests and treatments.

In this case, the main results of the evaluation are based on reports from a single informant from each clinic who reported whether the clinic had implemented each of the recommended practices within 6 months, but the evaluation failed to address whether the intervention effectively improved care. This approach also raises questions about the reliability and validity of such reports. Reliability is defined as the extent to which a particular characteristic (in this case, of a clinic) is measured consistently, whether, for example, as reported by different respondents (inter-rater reliability) or by the same respondent over time (test-retest reliability). Reports on organizational practices from a single informant tend not to be highly reliable unless the respondent is reporting on basic structural characteristics. For instance, in an earlier study, we found that a single respondent could accurately report the number of physicians in a practice, but in order to obtain reliable measures of clinic operational characteristics, more responses were required than there were members of a typical clinic.28 Thus, relying on a single informant often can be problematic, particularly for aspects of clinical management that might not be applied uniformly across a clinic.

Validity refers to the extent to which responses reflect what is actually happening in the clinic. In this case, there also were reasons to question the validity of the responses. Respondents participating in the QI program likely would have been biased to provide responses that conformed to the tenets of the educational program, whereas those who were in the control group would have been less likely to endorse these activities simply as a result of being in the program.29,30 Thus, this level of evidence might be suggestive of the success of the program, but should only be considered as hypothesis-generating, and more objective evidence would be required before concluding that the program was effective.

A better design would be to collect these data via other means. With sufficient funding, chart reviews could be conducted at both intervention and control sites. Alternatively, it also is possible that these data could be collected passively—in this case, through the VA health care system’s unified electronic medical record if one could identify the patients being seen in the clinics.31 In this case, the ultimate goal of this program is to perform laboratory tests and vaccinate patients as needed and to make sure that those who screen positive are reliably referred into care, all of which can be ascertained through the EMR. Either of these approaches would provide the strongest evidence of the success of the program. In addition, researchers also could passively access other potential SUD treatment sites to expand the candidate pool of control centers.

Text Box 1. Teaching Points for Quality Improvement Evaluations

As opposed to the controlled settings of clinical trials, most QI interventions are implemented in the “messy” world of actual clinical practice. As noted by others, QI interventions seek to change behavior rather than create new scientific knowledge, and learning in QI is driven by experiential learning.32 Moreover, improvement efforts are context-dependent, and both the interventions and the outcomes can be modified over time based on feedback.33 Thus, qualitative interview techniques such as those used in this study are well suited to learning about the nuances of an intervention and the potential barriers and facilitators to implementation. Thus, once it is known whether an intervention is effective, qualitative techniques are useful for understanding the how of the intervention—the mechanism, the specific populations, and the settings where it is most effective.

Such techniques also can inform efforts to redesign or improve interventions. Evaluations can be used to assess the extent to which clinics adhered to the intervention, and can potentially identify aspects of the program that were more or less likely to be adopted by participating clinics. For instance, in the current study, the interviews assessed whether the clinics addressed each of the recommended practices at 6 months. Thus, such interviews can provide data on how well the intervention was being implemented and aspects of the intervention that seemed to work better or worse. Such information would be useful for potentially redesigning the intervention.

In addition, qualitative interviews can be particularly effective for identifying barriers to implementation as well as aspects of clinics that might have facilitated the intervention. Such knowledge can be useful in propagating interventions to other clinics, whether by informing the selection process by focusing on clinics that lacked specific barriers or by helping to inform changes in the program that might overcome barriers or take advantage of facilitating factors. Issues such as the lack of time, staffing, or attention such as seen in this study might lead designers of interventions to require specific commitments of resources from senior management, whereas challenges such as poor collaboration with the hepatitis clinic might require active participation of health system leaders.

Qualitative interviews also can provide insight into how particular characteristics might have influenced the success of the implementation. In this study, participating clinics ranged in size (including both the number of staff and number of patients treated) as well as organizational structure, and some of these might have been more conducive than others to accomplishing the goals of the QI project. In particular, receiving support, including both resources and buy-in from the greater health system, was seen as important to the success of individual clinics.

Thus, understanding both whether an intervention was effective and how it worked are crucially important in implementation science. Both aspects are required in order to allow readers to understand how a particular intervention might work in their own context and to help guide the investment of their scarce resources.