A mixed methods descriptive investigation of readiness to change in rural hospitals participating in a tele-critical care intervention

South Carolina (SC) is a state in which rural communities disproportionately experience health care disparities [21]. Given severe resource limitations and a national shortage of critical care clinicians [22, 23], rural SC communities are unlikely to obtain critical care experts on site who can optimally manage severe sepsis and trauma. Thus, with external funding (NIH/NIMHD, RC1MD004405-01) the Critical Care Excellence in Sepsis and Trauma program (CREST) was established at the Medical University of South Carolina (MUSC). The overall objective of CREST is to improve the timeliness and quality of key treatment decisions in sepsis and trauma. To accomplish this, CREST partnered with four rural SC hospitals’ Emergency Departments to disseminate clinician education and telemedicine consultation services and strategies. CREST hospitals were chosen from a geographic region of SC along the Interstate-95 corridor which transects the state. This region of SC has marked health disparities including increased mortality from sepsis and trauma and a high proportion of minorities and citizens living below the federal poverty level. Hospitals were then selected for participation based on two criteria: designated as rural by the US Department of Agriculture and designated as fully medically underserved [21]. The sites have no formal affiliation agreement with MUSC. Together the 4 hospitals serve 165,500 citizens with 27% of the population living below the federal poverty level [24].

CREST was implemented in a stepwise fashion at partner hospitals in the following manner. First, administrative and regulatory issues were considered. This included memoranda of agreement (MOA), human subjects designation and IRB process, physician credentialing and privileging and explication of current and future financial obligations. Next an onsite clinical educational program in sepsis and trauma was conducted. The education included training in the use of the web-based telemedicine equipment. The final phase was activation and utilization of the telemedicine consultation service which was embedded into existing MUSC clinician on-call systems. During the grant period no reimbursement to the lead organization was required as it was covered by grant funds. CREST remains ongoing.

This is a multi-method, observational case study. There has been increasing research interest in mixed methods research as it provides a more comprehensive picture of health services than individual methods alone [25, 26]. We employed a core qualitative component as well as a supplementary quantitative component [27]. Data were collected during the implementation and medical education phases of the CREST intervention in 2010–2011. This study was approved by our Institutional Review Board which is responsible for overseeing the ethical conduct of human subjects research. Key informants gave signed informed consent prior to participation and a waiver of signed consent was granted for survey respondents.

Conceptual frameworks which integrate the theoretical domains of multiple disciplines are receiving increasing attention given convincing evidence of the need to understand the context in which change and adoption take place [14, 20, 28, 29]. In this study we adapted elements of an organizational readiness for change model proposed by Weiner [28]. We incorporated factors and items suggested by Stetler [29] and Helfrich and colleagues [18]. These included such factors as clinician perceptions about the evidence related to efficacy of telemedicine, and factors related to facilitation of adoption such as local champions and available resources. Other investigators have discussed related factors consonant with a broad readiness model [20]. For example, the Content, Context and Process Model of the strategic management of change [30] urges differentiation between non-receptive and receptive features of the organizational context. Hu and colleagues [9] purport that behavior potential is predicted by attitudes, perceived usefulness and perceived ease of use. In our observations and inquiries, we were also guided by the classic domains which affect quality of care, factors related to structure, process and outcome [31]. Alexander and Hearld [32] emphasize the need to study multiple perspectives on delivery system design and Weiner [28] urges that readiness items should focus on the specific impending change which we did in our data collection queries (i.e., reference to CREST). Broadly, our model included readiness for CREST, the dependent measure in this paper; and four potential mediating domains; historical contextual factors such as experience with host institution and other telemedicine projects; enabling generic contextual factors of the organization such as leadership, staff structure; and two CREST specific domains, “change valance” factors which could negatively or positively impact decisions to participate, such as perception of need, and awareness of scientific evidence and assessment of CREST characteristics, such as task demands and resources needed see Figure 1.

Three data sources were used: systematic project field notes, key informant interviews with CREST hospital leaders and selected staff, and a self-administered survey of clinicians at the four participating hospitals.

Key informant interviews (KIIs) were conducted to glean motivations and concerns of clinicians and administrators. KIIs were held primarily in person at the hospital, but occasionally by telephone because of difficult scheduling. Interviews lasted from 20–40 minutes, recorded with permission, and with back-up hand notes. We asked open ended items about perceptions of evidence, need, the hospital and Emergency Department context and facilitation elements [33].

The CREST clinician survey items were informed by the literature, investigators’ observations and KII data. Mindful of respondent burden, we adopted or adapted parsimonious dimensions of domains suggested by organization theory [28]. Items queried participants about perceptions, experiences and ratings, with a 6 response category choices ranging from 1–6 from “strongly agree” to “strongly disagree”. A choice of “don’t know/not applicable” was provided. Within the Hospital Context domain, three factors were queried: leadership and quality improvement (5 items), ED culture (5 items) and strategic concerns such as hospital transfers (2 items). Within the CREST Change Valance domain, defined as how staff value a specific impending change [28], three factors were investigated: evidence assessment (1 item), need (3 items), and usefulness (3 items). Within the CREST Information Assessment domain, 4 factors were queried: task demands (5 items), resource availability (3 items), involvement/communication (3 items), and evaluation needs (4 items). Change readiness was measured by one item “ED staff is receptive to CREST”. The CREST survey was self-administered anonymously at the start of the CREST educational session at each hospital.

Data from each source were first summarized for each hospital site; then responses were compared across sites. Qualitative data from the managerial reports and KIIs were sorted for themes related to the core domains and factors by two investigators (LH, JZ) and then reviewed by another investigator (DF) until consensus was achieved.

Given the small number of respondents, survey data were pooled across the hospitals. We reviewed the frequencies of responses to the individual items as we reviewed comments from the other two data sources [26]. Additionally, the 34 survey items were combined based on the content face validity and where they fit into the conceptual model domains. See Table 1 for the domains, factors and related items. This resulted in three summary scores where low scores indicated general agreement and high scores expressed disagreement. This was a crude method for exploring the extent to which survey responses agree with the hypothesized relationships posed by the innovation model. We used simple summation across responses and assigned a zero value to missing and not applicable responses. Thus, all factors had a possible low value of 0, but the high values varied. For example, a factor with three relevant items will have a maximum score of 18, while a factor with five items will have a possible score of 30. This process allowed the use of regression analyses to examine correlations between each factor and our proxy measure for commitment to CREST (ED staff are receptive to CREST). In the regression modeling, the Likert scale response to the dependent measure was used as a continuous variable. SAS 9.2 was used for survey data analyses [34].

Findings from all data sources were then triangulated or integrated [35]. While there are several operational definitions of triangulation [27], we use it here to provide rich data on potential content validity—that is reports of factors related to adoption. Additionally, we compare data from more than one source to confirm if an observation from one data source is evident in another data source [27]. In this exploratory research, we do not view any of the data sources as the “gold standard”, that is, we are not assessing criterion validity.

KIIs from two hospitals reported experience with telemedicine programs, notably an MUSC tele-stroke program (REACH) [36] and a SC Department of Mental Health (SC DMH) tele-psychiatry program [37]. It was evident that this experience enabled support for CREST. One KII commented that “REACH is the word. Everyone wants CREST because of the success of REACH”. Another noted that the quality of patient care had increased – length of stay, choice of medications, communication between ED physicians and psychiatrist, patient satisfaction—because of SC DMH Tele-psychiatry.

With respect to previous interaction with MUSC, some concern was expressed by interviewees from 3 hospitals about the importance of getting feedback communication on the patients transferred to MUSC. Two hospitals stressed the need to communicate processes of care with local primary care physicians. Distance to MUSC was mentioned several times. One asked if MUSC would resist accepting transfers of indigent patients.

KII and staff survey data both reflected important positive and negative factors which could affect readiness to be involved with CREST.

There was general agreement among KIIs at all hospitals on the need for CREST. Responses highlighted need because of lack of specialty staff and resources. Between hospitals however, there were differences of opinion on the consultation need for sepsis as compared with trauma. For example, KIIs from one hospital with a very large geriatric population emphasized their need for help with sepsis cases. Another felt great need for assistance with trauma cases as neurosurgical and orthopedic specialists were not available. In three hospitals, there was difference in KII rating of need by discipline—since ED physician staff was contracted and often changing, the nursing staff emphasized the need for CREST more so then did the physicians. The perceived need was corroborated by the survey item “the need for CREST is supported by your clinical experience here at your hospital” to which 78% responded they strongly agreed and another 13% moderately agreed (mean=1.36).

In the KIIs, there was next to no familiarity with or mention of the scientific evidence related to telemedicine efficacy. However 65% (mean=1.56) of survey respondents reported they strongly agreed that “the CREST program strategy is strongly supported by key evidence in the clinical literature”. KII comments indicated a disposition toward “common sense” observation that a resource for expertise was necessary and would be effective. With respect to “evidence”, interviewees at two hospitals again emphasized their experience with the REACH and SC DMH Tele-psychiatry programs.

When reflecting on potential usefulness of CREST, interviewees from all hospitals commented on the potential usefulness of CREST to improve the quality of care for patients, “quicker” consults and better outcomes. When queried about patient/family acceptance, all KIIs felt that the response would be very favorable: “patients will be amazed”; “it will be seen as a blessing.” Survey respondents reported that CREST would enable them to complete patient care more quickly, would improve productivity and enable them to improve quality care 42% (mean=2.05), 45% (mean = 1.68), and 52% (mean=1.99), respectively, strongly agreed.

The structural features of the hospital’s organization were highlighted by KIIs as well as by CREST investigators’ notes. Most frequent were comments on availability of specialist staff. In one hospital, administrative issues were difficult with leadership changing three times among contracted management firms and because of poor physical plant for administration and the ED itself. Interviewees at several hospitals pointed out that the ED physicians, including the Medical Director were not hospital employees, but rather contracted physicians and frequently were hard to reach and/or had limited hours.

With respect to leadership and the structure and processes of their quality improvement activities, different features were highlighted. High-level leadership was at least nominally supportive, and was highly present in two of the hospitals according to both KIIs and project management notes. The QI presence was quite variable. At only one hospital was a structured QI unit championing CREST and coordinating the planning and implementation process. The responses to survey items related to hospital leadership culture (not necessarily focused on CREST) demonstrated less agreement that “hospital leadership encouraged employees to take risks and be innovative” (9% strongly agreed) or promoted team building to solve clinical care problems at the hospital (23% strongly agreed). In fact the survey items about hospital leadership quality had higher means, i.e. stronger disagreement, than all the other groups of items.

The observations in the staff field notes about communication processes were noticeably different among the hospitals. For example at one hospital, the medical staff and the ED staff appeared in regular communication, not so at other hospitals. The ED culture at two hospitals appeared as functional and organized—as illustrated by survey comments about nurse-physician collaboration -- the nursing staff as “willing to learn” and all clinicians as “responsive to change”. Championship by ED directors varied from “being distant” to being committed to communication and sharing. Two camps were evident in KII statements at one site, “enthusiastic young Turks and the old guard”. Survey items referring to the ED culture were rated generally in agreement but not strong agreement. For example, approximately 25% of respondents strongly agreed that “ED staff (MDs, RNs and other) cooperate and improve effectiveness of patient care”, and “ED staff have a sense of personal responsibility for improving patients care and outcomes”. The KIIs and investigator observations however frequently highlighted the importance of nursing staff leadership in day to day functioning.

Several strategic concerns were evidenced in the KII reports and also in the interactions with CREST investigators. Concern about “patient stealing” was heard from only one hospital. More common were the remarks about CREST helping with hospital survival (two hospitals), improving public confidence and trust and improving statistics with state regulators.

Interviewees offered many comments on CREST task demands and resource availability. A KII from one hospital stressed that the technology would have to be easy to use. One wondered if the tasks would result in increased time to disposition. Participants from all hospitals stressed that the education/training would be very important; “…training will make it or break it as there are many part-timers who are at some distance away”. All interviewees from one hospital in particular commented on the lack of space in the ED. KIIs at one hospital discussed their close collaboration with the local area EMT service as a strength in its collaborative network.

Several survey items queried about CREST resource assessments. Sixty one percent (mean=1.62) strongly agreed that implementation was feasible at their hospital, but only 32% (mean=2.32) felt their roles/responsibilities were clear; 48% (mean = 1.81) strongly agreed CREST would require ED staff teamwork. About one-third of respondents strongly agreed there were enough resources and enough nurses to implement CREST (2.32 and 2.18 respectively); 40% (mean = 1.97) strongly agreed there were enough MDs for implementation.

KIIs were asked about their involvement and communication among personnel to date. Response varied but most felt they were at least somewhat informed. At two of the hospitals, KIIs referred to particular “champions”. In two hospitals, comments about the challenges of communication were voiced by CEO/senior administrator and the nurses as there was no real ownership at the ED level. In the survey, while 36% (mean= 2.12) of respondents strongly agreed communication with staff had been excellent, 27% (mean=2.85) strongly agreed they had been appropriately involved. The reality of facilitating the implementation of CREST was most evident in the structured notes of the staff and investigators. Managerial notes chronicled the interactive processes of developing and signing a MOA, applying for Federal Wide Assurance as required in human subjects research, and establishing credentialing and privileges. The time and energy needed by CREST staff was judged to be directly related to the organization's available support staff and efficiency. Time for completion of administrative tasks ranged from 11–12 months, and would have actually been longer in one or two without diligent administrative prodding.

With respect to evaluation expectations and availability of metrics for success, several KII comments were offered. Most common was the mortality rate pre- and post- CREST implementation, particularly for sepsis. Getting feedback on transferred patients from MUSC was again mentioned. In the survey, more than half strongly agreed that collecting feedback from staff, developing and distributing CREST performance measures to staff and providing a forum for discussion of implications for continued improvement.

The survey responses to the item “ED staff is receptive to CREST” were 42% strongly agree, 23% moderately agreed, 32% agreed, 2% disagreed and 1% moderately disagreed (n=20 missing).

Table 3 reports the exploratory findings from the regression model of independent associations of the summary measures with the rating of receptivity to CREST. The results of the five-factor model (Model 1) show that the factors were not equal in explanatory power. The summary measure of perceived need for CREST is the most important factor in explaining the variations in CREST receptivity. The summary measure of hospital leadership and QI culture had the lowest explanatory power and removal of this factor from the model improved explanatory power. The factors that measure aspects of ED culture also contributed little to explaining variation. This factor contains items related to cooperation between hospital administration and ED leaders to make needed changes, ED staff responsibility for care quality, cooperation between ED staff, and willingness of ED staff to try new approaches.

The three factors that explain nearly 50 percent of the variation in responses to the change commitment measure are “need for CREST”, CREST “task demands” and perceived “resource needs” relevant to implementation. Respondents who strongly agreed to the “task demands” items also strongly agreed that ED staff is receptive to CREST. Respondents who strongly agreed that CREST is feasible, the technology is easy to use, their roles vis-à-vis CREST are well defined and ED teamwork is required for CREST to succeed, also recorded a strong CREST commitment score. Individuals who strongly agreed to the “resource perception” questions (we have enough physicians and nurses and general resources to implement CREST) also reported strong CREST receptivity scores.