Learning from national implementation of the Veterans Affairs Clinical Resource Hub (CRH) program for improving access to care: protocol for a six year evaluation

Published protocols for large-scale program evaluations have the potential to improve the quality and usefulness of subsequent evaluation results [1]. This paper describes the evaluation protocol for assessing national implementation of the Department of Veterans Affairs’ (VA’s) innovative Clinical Resource Hub (CRH) program.

In brief, CRH aims to improve patient access to local outpatient care by implementing regionally organized contingency primary care and mental health staffing support to outpatient care sites, primarily through telehealth modalities. By contingency staffing, we mean time-limited staffing to cover staffing vacancies. To accomplish this, CRH aims to implement 18 regional VA hubs that employ contingency staff to remediate vacancies across VA’s more than 1000 outpatient sites, using primarily virtual care modalities [2]. While temporary employment agencies have long been used by industries to meet short-term staffing needs, and more recently have been used to address temporary nursing [3] or physician [4, 5] staffing gaps, we found no prior scientific publications describing healthcare organization implementation or evaluation of large-scale, internally staffed, regionally organized contingency staffing support for geographically dispersed outpatient sites. The CRH evaluation focuses on assessing whether and how CRH achieves its implementation and outcome goals.

The VA healthcare system is one of the largest managed care systems in the United States, providing care nationally to over 9 million military Veterans across all 50 United States and US territories [2] under the direction of the United States Congress [6]. The VA system provides comprehensive health services including primary care, mental healthcare, and specialty care through a regional administration system of 18 Veterans Integrated Service Networks (VISNs) that operate 171 medical centers and more than 1000 Community Based Outpatient Clinics (CBOCs). Medical centers manage far-flung networks of CBOCs, rehabilitation units, nursing homes, and other facilities [2]. VA is a learning organization and as such funds a large health services research program that includes investigator-initiated research, quality improvement research [6] and VA operations-partnered research [7]. Among othe partnership research priorities, VA aims to rigorously evaluate all new national programs including CRH.

The CRH program was initially implemented through VA-wide policy change. In 2018 the United States Congress mandated that VA focus on improving access to care through better support for staffing deficits [8], as well as through other program improvements such as increased access to non-VA community resources [2]. In response, among other improvement initiatives, VA leaders initiated CRH planning. In October 2019, the VA Office of Primary Care (OPC) began implementing CRH, based on a directive to all 18 VA administrative regions (Veterans Integrated Service Networks or VISNs) and their over 1000 VA outpatient sites. The directive built on three prior pilot projects that tested providing regional telehealth staffing support, [9‐12] as well as on existing VA telehealth capabilities [9]. In August 2020, OPC requested that its Primary Care Analytics Team (PCAT), an embedded research and evaluation unit, [13] plan a comprehensive six year evaluation of the national CRH program.

Publication of protocols for program evaluations is less common and less standardized than publication of protocols for randomized trials but may be increasingly important as healthcare delivery systems and their funders take steps to become learning health systems [14, 15]. Our protocol development process aimed to foster scientific consensus across multidisciplinary researchers as well as tightly connecting researcher and clinical operations partner goals. We expect that our protocol will continue development over the duration of the project as a living document that serves these needs.

The CRH evaluation is built on the initial CRH program goals, rationale, implementation strategies, and structure, and calls for documenting these over time. The protocol also calls for multi-method evaluation of the types, quality, and effectiveness of CRH services, as well as their cost and acceptability to stakeholders, and includes an initial formative evaluation focus followed by a summative evaluation of program effectiveness and value.

The evaluation protocol relies on the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) [16, 17] framework as the foundation for a logic model, specific aims, evaluation questions, and data collection and analysis plans.

The objectives for developing the CRH evaluation protocol were to prospectively: 1) identify the outcomes CRH aimed to achieve, and the key program elements designed to achieve them; 2) specify evaluation designs and data collection approaches for assessing CRH progress and success, including return on investment relative to program goals; and 3) guide the activities of five geographically dispersed evaluation teams.

This paper describes the protocol for evaluating a nationally implemented, regionally based VA contingency staffing program for improving patient access to care. The program aims to provide time-limited primary care and mental health staffing, primarily through telehealth modalities, in support of over 1000 VA outpatient sites. The planned evaluation addresses the dearth of studies on contingency healthcare staffing support by describing and assessing the program’s implementation and outcomes.

Health services researchers are increasingly asked to evaluate newly implemented healthcare delivery system programs or policy changes [13, 53]. Large scale implementation is often driven by organizational or political priorities, such as those identified by the US Congress in the case of CRH, and by associated tight timelines that prevent full program or evaluation design prior to implementation. Rigorous program evaluation has the potential to support improved implementation within learning healthcare systems [6, 7, 13, 14] by providing both data showing needed program course corrections during implementation and data aimed at supporting evidence-based decision-making about whether or how program should be maintained once implementation is complete. Yet while extensive guidance on developing protocols for tightly controlled studies such as randomized trials abound, practical guidance on exactly how to develop a protocol for a new and evolving health system-driven program is uncommon [54, 55]. Our protocol development process aimed to use implementation science frameworks to integrate program stakeholder perspectives into a scientifically valid and feasible learning health system-oriented evaluation plan.

We expect that publication of our protocol will maximize the validity and utility of our conclusions as documented in subsequent CRH evaluation publications. Both formative and summative evaluation components emphasize goals, concerns and primary and secondary outcomes identified by CRH program leaders and stakeholders during the initial CRH implementation period. The protocol can thus help the evaluation to maintain a strong focus on what program developers and implementers specified as their intended accomplishments.

Our protocol addresses contingency staffing issues that, based on literature review and early input from CRH implementers, are likely to affect any contingency staffing initiative. We therefore expect that our systematic protocol development process will have supported the relevance and comprehensiveness with which we can inform VA and other learning healthcare systems about contingency staffing in healthcare. For example, our literature review showed that provision of virtual care from staff hired from outside of a local site may be attractive from a variety of perspectives. However, potential negative impacts on patient care and workplace outcomes can occur [56]. Negative impacts could occur in CRH due either to lack of integration of hub contingency staff into spoke site supervision and workflow, or to the substitution of telehealth visits for needed in-person care [57‐59]. The protocol calls for data to address these issues, including documenting the type and duration of staffing support provided.

We recognize limitations inherent in our plan. First, our non-randomized study design will not yield definitive information about a causal link between implementation of the CRH intervention and the summative outcomes we identify. Second, in an evolving and growing program, there will ongoing evolution of key data elements. To address this, we will define alternative variable definitions and conduct sensitivity analyses when indicated. We will also account for pandemic-related changes in overall VA use of virtual care by accessing national VA data bases from 2017 onward for CRH supported and non-CRH supported sites. Third, although we will account for key characteristics of sampled sites, staff and Veterans, our ability to assure site-level equivalence for quantitative analysis or for developing representative samples for qualitative work will be limited. Fourth, the evaluation engages Veteran perspectives through qualitative interviews and surveys, rather than direct participation, until 2025, when we will engage them in a panel for considering summative evaluation data (see Table 1). Fifth, early CRH evaluation data suggests that there may be differences between mental health and primary care telehealth modality use [60]. We will address these and other important differences across the two types of services through further protocol development. Sixth, our evaluation focuses on the primary care and mental health components of CRH. These components were the first to be implemented. Engagement of other specialties as part of the CRH program is planned for the future and will be the subject of a separate program planning and evaluation effort. Finally, additional challenges are sure to arise as we implement the evaluation plan. We will account for these in study records and in relevant publications. Assessment of implementation strategies used and of efforts related to CRH sustainment are two areas in which we expect to further develop data collection and analysis as the CRH program matures.

We also recognize potential strengths of our evaluation. By showing how researchers and health systems leaders partnered to integrate the aims, capabilities, and perspectives of each into a final protocol, this paper supports continued development of approaches for achieving learning health systems goals in VA and other large healthcare systems. The protocol development process used a systematically applied researcher and clinical operations partnership model to ensure coverage of both scientific and stakeholder goals. Use of implementation science frameworks, data triangulation from multiple sources, and systematic transition from a formative to summative focus may enable program improvement while driving toward final health system decisions. Finally, engagement of evaluation teams with different focuses during protocol development enables national scale data collection and analysis while the engagement of a coordinating core team maximizes evaluation coherence and synergy.

In summary, we developed an evaluation protocol, informed by implementation and contingency staffing theory, in partnership with delivery system and program leaders who are committed to the concept of VA as a learning organization. Publication of our protocol is aimed both at promoting the integrity, validity, and usefulness of our findings and at sharing our lived example with others interested in partnership evaluation in real-world settings.