The organizational social context of mental health services and clinician attitudes toward evidence-based practice: a United States national study

Multiple factors at system, organizational, and individual levels influence the implementation of evidence-based practices (EBPs) and other innovations in public sector social service and mental healthcare settings [1‐4]. These factors include policy, social and economic characteristics, characteristics of the innovation itself, characteristics of the organization attempting to implement the innovation, and characteristics of service providers and clients [1, 3‐8]. One of the most proximal influences on service providers’ attitudes and behaviors is the social context of the organizations in which they work [9]. This paper examines the association of organizational social context, other organizational factors, and clinician demographics with clinician attitudes toward adopting EBPs in a United States national sample of mental health clinicians.

The role of organizational social context in the development of individual-level work attitudes and behavior has been the focus of organizational research for several decades. This long line of research has established that work-related attitudes of individuals in a variety of different types of work environments are influenced by the organizational social contexts in which they work [5, 10‐12]. Organizational social context includes the norms and expectations (i.e., culture) of the organization for its members as well as the psychological impact of the work environment on the individual workers (i.e., climate). Culture and climate are related, complex, multidimensional constructs, but there is evidence that they are distinct and that each affects work attitudes in unique ways [13]. Culture is expected to influence the work attitudes of individual workers through their accommodation to the expectations (e.g., an emphasis on proficiency) that govern their day-to-day work behavior. Climate is expected to influence work attitudes through the psychological impact of the work environment (e.g., the level of stress) on the workers.

While the vast majority of research on culture and climate has been conducted in business and industrial organizations, there is a growing interest in the cultures and climates of healthcare, mental health, social service, and other human service organizations. While few controlled studies have examined actual organization context change efforts [14], research provides evidence that culture, climate, and work attitudes compose an organizational social context that is associated with worker turnover, new program sustainability, service quality, and service outcomes [9, 13, 15‐20]. Here, we examine variance in mental health clinicians’ attitudes towards EBPs explained by specific dimensions of organizational culture and climate in order to better understand the role that organizational social context might play in complementing or inhibiting efforts to disseminate and implement EBPs.

Mental health clinicians have varying and complex attitudes toward EBPs [21, 22]. While an overall positive or negative view of EBPs may be held by clinicians, clinicians can also simultaneously hold somewhat contradictory attitudes. That is, clinicians can be positively predisposed to EBPs on one dimension and negatively predisposed on another. For example, the idea that the practices clinicians use in providing mental healthcare should be informed by empirical evidence might be appealing to some clinicians who might also be unwilling to implement specific EBPs that are required by their employer or by a state mental health agency [23]. Or, clinicians may be open to learning more about EBPs, but at the same time believe that what they have learned in the world of practice experience is much more salient to selecting appropriate psychotherapeutic approaches than evidence provided by published research [23].

Aarons et al. [21, 23, 24] captured this complexity in the Evidence-Based Practice Attitude Scale (EBPAS) and documented four distinct dimensions in clinicians’ attitudes toward EBPs. The four dimensions include the intuitive appeal of EBPs, the likelihood of adopting EBPs as a result of institutional requirements, the perceived divergence between research-based practices and current practice, and general openness to learning new practices. Moreover, they found that certain clinician characteristics at the individual level (e.g., years of experience, education level) influence each of these various dimensions differently. For example, clinicians with more advanced degrees described EBPs as more appealing but at the same time were less willing to implement EBPs simply because they were required.

EBPAS scores have been associated with provider characteristics related to implementing new service technologies (See Aarons et al., 2012 for a review [25]). For example, female service providers have higher scores on the Appeal [26, 27] and Requirements subscales [26] as well as higher total EBPAS scores [26]. Higher educational attainment has been associated with higher scores on the Appeal subscale [23, 26‐28] and lower scores on the Requirements subscale [26]. Aarons [23] found no difference in EBPAS scores by discipline (e.g., social work, psychology), but a later study found that providers with their highest degree in social work had higher Openness subscale scores and higher Total EBPAS scores than those with their highest degree in psychology. Clinicians with a degree in psychology, however, endorsed lower perceived Divergence between EBP and usual practice compared to clinicians with a degree in social work [26].

EBPAS scores have also been associated with knowledge and use of EBPs in other healthcare disciplines and settings. In a study of physicians, more positive EBPAS scores were associated with higher levels of physician knowledge of technological innovations supporting evidence-based clinical practice [29]. In a study of implementation of Parent–child Interaction Therapy (PCIT), more positive scores on Appeal were related to increased clinician attendance in PCIT consultation, while lower perceived Divergence between EBP and usual care was associated with increased phone coaching [30]. In substance abuse treatment settings, more positive attitudes toward EBP have been demonstrated to predict self-reported use of evidence-based cognitive/motivational treatment approaches [31] and adoption of contingency management, an evidence-based substance abuse treatment approach [32].

Studies examining transfer of learning in varied settings and types of health practice have also found relationships between attitudes and EBP. One study found that occupational therapy students in Ireland endorsed willingness to practice EBP [33]. Another study found that while medical residents in Japan endorsed generally positive attitudes toward EBP, few could articulate the construct or were familiar with the use of standard research sources in medical decision making [34]. However, a survey of physicians in the United Kingdom found a greater prevalence of positive attitudes toward EBP and greater ability to articulate elements of evidence-based medicine [35]. Thus, while studies across countries and healthcare practice settings have indicated variability in attitudes and knowledge related to EBP, none have examined the link between organizational context and attitudes in nationally representative samples.

Studies have indicated that change in organizational culture and climate in healthcare settings is a complex proposition that must take into account the impacts of autonomy and influence that occur in complex healthcare organizations [36, 37]. In addition, individually held attitudes constitute but one factor in a complex context that may influence adoption and use of EBP [6]. Still, providing conceptual linkages between system and organizational context, provider attitudes and characteristics, and adoption and use of EBP is needed to help understand and develop strategies to improve EBP implementation and sustainment.

The organizational social contexts of providers’ work environments are also related to providers’ attitudes toward EBP. Positive climate measured by the Organizational Readiness for Change scale was negatively related to Divergence scores on the EBPAS [38], and demoralizing or negative organizational climate has been found to be positively related to Divergence scores [15]. These results indicate that providers in organizations with positive climates perceive less Divergence between EBP and their usual practice, while those in organizations with poor climates perceive more Divergence. An organizational culture that promotes achievement and mutual encouragement has been positively associated with higher scores on Appeal and Openness subscales, as well as higher total EBPAS scores, while an organizational culture emphasizing rules and orders (i.e., ‘command and control’) over personal beliefs and ideas, has been associated with higher Divergence scores [15]. Finally, mental health providers working for private non-profit organizations endorsed more positive attitudes toward EBP relative to public sector mental health departments [39].

Previous studies linking organizational social context and attitudes toward EBPs were conducted with limited samples and fewer organizational context dimensions, so further validation with more representative samples using multidimensional measures of both organizational culture and climate is needed. The present study builds on previous work by examining the association of attitudes toward EBP as measured by the EBPAS [23, 26], with multiple dimensions of organizational social context as measured by the Organizational Social Context (OSC) scales [16] in a nationally representative sample of mental health agencies from all regions of the United States.

We developed several hypotheses based on the extant literature. By including predictors at both organizational and individual levels in a multi-level analysis, it is possible to partition the variance in attitudes explained by factors at each level. This is important because we know that culture and climate characteristics at the organizational level are not completely independent from individual clinician level characteristics. For example, organizational policies and leader behaviors can signal to providers the important cultural values of the organization [40]. Therefore, analyses at a single level—whether the organizational level or individual level—could confound the association of attitudes toward EBPs with variables at the other level. In addition, it is important to consider the independent and combined roles played by multiple dimensions of culture and climate simultaneously in explaining different dimensions of clinicians’ (often conflicted) attitudes toward EBPs. Thus, we hypothesized that variance in attitudes toward EBPs would be explained by variables at both the organizational and individual levels.

Organizational culture and climate are each expected to influence attitudes about EBPs in different ways. Organizational culture captures the expectations and values about what is important in a specific organization. These expectations and values, either implicitly or explicitly expressed in the behavior of fellow workers, have the capacity to socialize members of the organizations who may seek to behave in ways that meet the expectations of their workplace [9]. As a result, culture should influence attitudes about EBPs in ways that comply with organizational expectations. For example, clinicians who work in cultures that expect organizational members to be proficient in their work (i.e., have up-to-date knowledge, be effective in their work) should have more positive views of EBPs. Thus, we hypothesized that providers working in organizations with proficient cultures would have more positive overall attitudes toward EBPs as evidenced by higher scores on the EBPAS total scale score. In addition, we hypothesized that more proficient cultures would be associated with higher scores on the openness, appeal, and requirements scales, and lower scores on the divergence scale. We hypothesized that clinicians working in organizations with rigid or resistant cultures would be more likely to have lower EBPAS total, Appeal, Openness and Requirements scores, and higher Divergence Scores.

Organizational climate is expected to influence attitudes about EBPs somewhat differently. Climate describes the psychological impact of the work environment on clinicians (e.g., stress) [9]. Therefore, although an organization might expect its members to be proficient, clinicians in high-stress work environments might feel that requirements imposed on them by implementing new EBPs are overwhelming. Thus, we hypothesized that providers working in organizations with more stressful climates would endorse more negative attitudes toward EBPs as evidenced by lower scores on the Appeal, Openness, and Requirements scales, and higher scores on the Divergence scale. We also hypothesized that clinicians working in organizations in more functional climates would report more positive attitudes toward EBP. Finally, we hypothesized that clinicians working in organizations with engaged climates would endorse higher scores on the Appeal scale because this scale emphasizes the importance of colleague perceptions and practicality of EBP.

The national onsite, in-person survey of mental health clinicians who provided mental health services in the sampled clinics is one of two surveys composing the Clinical Systems Project (CSP) of the Research Network on Children’s Mental Health funded by the John D. and Catherine T. MacArthur Foundation [16]. The other survey was a telephone survey of mental health clinic directors [41]. The sampling frame for these surveys began with the counties selected for the National Survey of Child and Adolescent Well-being (described in [42]). The sample of 100 clinics participating in the clinician survey represents clinics from the CSP that met minimum size criteria (five or more clinicians) and whose director agreed to an onsite survey of clinicians in a scheduled staff meeting. Comparisons of the characteristics of clinics represented in the clinic directors survey that did and did not meet these criteria show similar characteristics in the number of clinicians employed in the clinic, clinician turnover rates, and the proportions of clinicians who are psychiatrists, Ph.D. psychologists, and MSW social workers. However, a slightly higher proportion of therapists in the clinics that participated in the clinician survey were BSW social workers (13% versus 8%).

The correlation matrix in Table 3 shows that the clinic response rate (i.e., the proportion of clinicians completing scales within each clinic) is unrelated to the OSC or EBPAS scale values, as indicated by correlations that range from −0.05 to 0.03 (all non-significant). The pattern of correlations within the matrix also suggests that the responses to the OSC and EBPAS are not explained by common method error variance. The size of the correlations between scales from the two instruments (OSC and EBPAS) range from a low of 0.00 (p < 0.990) for rigid culture and EBPAS total to a high of 0.24 (p < 0.000) for proficient culture and EBPAS total. Moreover, despite the statistical power of the large sample (n = 1,112), only half of the 35 correlations between the two sets of scales are statistically significant. Among the dimensions of culture and climate, proficient culture is the most highly correlated with attitudes toward EBPs. Significant correlations include the relationships between proficient culture and the EBPAS Total (0.24, p < 0.001), Requirements (0.19, p < 0.001), Appeal (0.20, p < 0.001), and Openness (0.20, p < 0.001).

The within clinic agreement in clinician descriptions of their clinic’s culture and climate was assessed with r_wg[45]. The average r_wg for each scale ranged between 0.91 and 0.96 for each clinic. The lowest r_wg values for each scale across all clinics ranged between 0.58 and 0.87, and the highest values for each scale ranged between 0.98 and 0.99. These statistics represent strong within clinic agreement among clinicians completing the OSC and support the aggregation of those responses for clinic level measures of organizational culture and climate.

Intraclass correlation coefficients (ICCs) in Tables 4 and 5 provide an estimate of the degree of clustering of responses to the EBPAS scales within groups (represented here by mental health clinics). Although ICCs are typically small in relation to estimates of the proportion of between group to total variation provided by an ANOVA-based eta-squared coefficient, the effect of clustering with ICCs as low as 0.01 to 0.05 can be dramatic on estimates using ordinary least squares regression that assumes ICC = 0.00 [50]. The ICCs reported in Tables 4 and 5 for the five EBPAS scales range between 0.02 to 0.06, and four of the five are statistically significant as reported in Table 5, indicating that attitudes about EBPs are clustered by clinic and thus need random regression models that account for the clustering. The ICCs and eta-squared coefficients (which range from 0.11 to 0.15) reported in Tables 4 and 5 confirm that a significant proportion of the variance in clinicians’ attitudes toward the use of EBPs is a function of the clinic in which the clinician works. Tables 4 and 5 also show the proportions of the organizational-level or clinic variance in attitudes toward EBPs that is explained by organizational social context and by all organizational and individual predictors combined.

The correlations in Table 3 provided evidence that the relationship between clinician-level responses to the scales measuring proficient culture and the total EBPAS was stronger than the relationship between any other pair of OSC and EBPAS dimensions. The higher order analysis in Table 4 provides further support for our hypothesis that clinicians who work in organizations with proficient cultures have more positive attitudes toward the use of EBPs. The results shown in Table 4 describe a significant unique positive effect of proficient culture on attitudes toward EBPs after controlling for other dimensions of organizational culture, dimensions of organizational climate, type of agency, service structure, clinician level perceptions of culture and climate, and other clinician characteristics (e.g., age, experience, education, et al.). Clinicians who work in clinics that expect them to place the well-being of each client first, to be competent and to have up-to-date knowledge are more likely to have more favorable views toward using EBPs than clinicians who work in clinics that do not have those expectations, regardless of their own demographic characteristics, experience, or training.

It’s important to note that this result was obtained after controlling for type of agency (for profit, non-profit, or public), clinic service structure (child and adult, child only, or separate child division), and clinician gender, education level, years of experience, race, profession (education, medicine, nursing, social work, other or psychology) and individual level responses to the culture and climate scales. Although not shown here, the same association between the clinicians’ attitudes toward EBP and their organizational culture was confirmed without the additional covariates in the analysis.

The HLM model shown above was repeated with each of the EBPAS first order scales (Appeal, Requirements, Openness, Divergence) and is shown in Table 5. Clinicians in clinics with proficient cultures report higher overall EBPAS Total and lower Divergence scores. In contrast, Organizational Climate has a stronger association with the EBPAS Appeal scale. In more engaged climates (where clinicians share the perception that they are able to personally accomplish many worthwhile things, remain personally involved in their work, and sustain concern about their clients), clinicians indicate they would be more likely to adopt an EBP if it were intuitively appealing, could be used correctly, or was being used by colleagues who were pleased with it. Clinicians in more stressful climates (those work environments where clinicians shared the perception that they are emotionally exhausted from their work, overloaded in their work, and unable to get the necessary things done at work) report that they would not be as likely to use an EBP if it were required by their agency, supervisor, or state. This suggests that formal requirements for using EBPs would not be as successful in organizational climates characterized by high levels of stress and that other, or additional implementation strategies should be considered.

The analyses also show that the service structure of the clinic was not associated with the clinicians’ attitudes toward EBPs but the type of agency did play a role in three of the five EBPAS scales (Total, Appeal and Openness). Clinicians in private for profit clinics generally had more negative attitudes toward the use of EBPs.

As shown in Tables 4 and 5, the dimensions of organizational culture and climate together explained the highest proportion of organizational variance in the EBPAS dimensions of Appeal and Openness. In addition, culture and climate together also accounted for the majority of the organizational variance that was explained by the full model in these two criteria and in the EBPAS Requirements scale.

At the individual level, several clinician characteristics were associated with attitudes toward the use of EBPs with years of experience, race/ethnicity and academic major having consistent associations across all five EBPAS scales. Clinicians with more years of experience and African American clinicians reported more negative attitudes toward EBPs after controlling of all other characteristics at the clinician and clinic level. Although the effects of academic major were not entirely consistent across all dimensions, social workers generally held the most positive views of EBPs as indicated by three of the five EBPAS scales.

The important finding in this study is that mental health clinic organizational culture and climate are associated with clinician’s attitudes toward adopting EBPs, even when controlling for the effects of a variety of organizational-level and individual-level characteristics. Clinicians working in organizations characterized by proficient cultures, engaged climates, and less stressful climates have more positive attitudes toward the use of EBPs, but each dimension of culture and climate is uniquely associated with a different dimension of the clinicians’ attitudes. The best unique predictor among the culture and climate dimensions of clinicians having positive overall attitudes toward EBPs is working in a proficient organizational culture. Clinicians in proficient cultures (i.e., the organization expects them to place the well-being of the clients first, to be competent, and have up-to-date knowledge) endorsed more positive overall attitudes toward adopting EBPs and lower perceived divergence between ‘in the trenches’ clinical work and the use of EBPs.

Two dimensions of organizational climate were associated with clinician attitudes toward EBPs, but each was associated with a different attitudinal dimension. Clinicians working in clinics with more stressful climates were less likely to adopt an EBP if it was required by a supervisor, agency, or state. This means clinicians working in settings with high levels of emotional exhaustion and role overload are less likely to respond favorably to policy or regulatory mandates requiring the use of EBPs.

Clinicians working in engaged organizational climates where there is a sense of personal accomplishment, personal involvement in and concern for clients reported a greater likelihood of adopting an EBP if it fit their views of clinical practice and their ability to effectively learn and use (i.e., to have high self-efficacy) a given EBP.

Given our conservative approach of testing hypothesized relationships after controlling for the effects of all other dimensions and predictors at the organizational and individual levels, a number of our hypotheses were not confirmed. The HLM analyses provided tests of the unique effects of each dimension of culture and climate at the organizational level after controlling for the effects of all other culture and climate dimensions and all other predictors at both the organizational and individual levels. This is important because all dimensions of culture and climate were related to at least one dimension of the EBPAS in the zero-order correlations shown in Table 3. Moreover, all dimensions of culture and climate at either the organizational or individual level were related to one or more dimensions of the EBPAS in the HLM analysis. However, our hypotheses and findings focus on the independent association of each of the organizational level dimensions of culture and climate with individual level attitudes toward EBP.

In regard to organizational culture, there were no significant organizational level findings related to rigid or resistant cultures (when controlling for the expectations associated with proficient cultures, the three climate dimensions, and all individual-level predictors). Rigid and resistant cultures were associated with attitudes toward EBP at the zero order level, but these relationships were attenuated when the other dimensions of culture and climate were included in the analyses.

Some of our climate related hypotheses were also not supported. While stressful and engaged climates were associated with poorer attitudes toward adopting EBP, our findings illustrated that only two EBPAS subscales were related to climate. Moreover, although functional climates were associated with EBPAS total, Requirements, Appeal, and Openness scales at the zero-order level, all of these relationships were not statistically significant when other dimensions of culture and climate were included at the organizational and individual levels.

It is possible that additional effects could be found if we utilized profile analysis to characterize clinics by culture/climate profiles (e.g., depicting the levels of culture and climate dimensions in relation to each other) rather than assessing the unique contributions of individual culture and climate dimensions after controlling for the others. That is, there may be profiles of positive and negative culture paired with positive or negative climate that provide a more complex characterization of organizations and are associated with EBP attitudes. Thus, future studies should examine profiles of organizational context that may be associated with attitudes and adoption and use of EBPs.

While organizational culture and climate and attitudes toward EBP have been explored in a previous study [15], the present study is the first to focus on these aspects of the complex relationship between multiple dimensions of social context and attitudes. However, it should be noted that the previous study utilized an earlier measure of organizational context with some overlap with the present measure of organizational social context. The present study advances this work by utilizing a more refined measure of organizational social context validated in a United States nationally representative sample of mental healthcare providers. In particular, the conceptualization of the culture and climate dimensions have been honed to be most relevant to health and allied health service settings and predictive of clinical outcomes and EBP implementation [50].

These findings suggest that the successful dissemination and implementation of EBPs among mental health service providers can be facilitated by the support of strategies for developing positive organizational cultures and climates that might otherwise present barriers to adoption and implementation [50]. In addition to implementation of clinical interventions, broad changes are now occurring in behavioral health settings with significant incentives to deploy accountability systems tied to monitoring and reimbursement. Such initiatives may impact organizational context or conversely be affected by organizational context. Changing organizational culture and climate in mental health and social service systems to prepare for implementation requires strategic, well-planned, and evidence-based strategies specifically designed for those systems [1, 40, 51]. Some promising developments in this regard are being tested in such organizations using randomized controlled trials, showing that organizational interventions can create improvements not only in organizational social context, but also in the implementation of EBPs and client outcomes [50, 52]. In related work, effective leadership and coordinated organizational strategies to develop a positive work environment have been suggested to promote strategic cultures and climates that support the implementation and use of EBPs [40, 53, 54].

Beyond organizational culture and climate, we identified other factors associated with attitudes toward EBPs that are important to understanding barriers to dissemination and implementation. The service structure of the clinic was not associated with clinician attitudes toward EBPs but the type of agency did play a role. Clinicians working in private for-profit clinics reported more negative attitudes toward the use of EBPs. The finding that clinicians working in private for-profit clinics reported more negative attitudes toward EBPs highlights the complexity of supporting EBP implementation and use. private for-profit clinics generally operate under different financing and reimbursement structures and processes such as fee-for-service. This is in contrast to private non-profits that may rely more heavily on contracts and grants to provide funding for services. The links between organization type and attitudes toward EBPs was explored in another study that found that clinicians working inprivate non-profit clinics, compared to public sector clinics, had more positive attitudes toward EBPs [39]. The authors attributed the finding to the more bureaucratic, hierarchical, and less flexible nature of programs that are part of large public sector mental health authorities.

At the individual level and consistent with previous studies, we found that female clinicians and social workers were more positive in their assessments of the use of EBPs while clinicians with more years of experience were more negative in their assessments. Further research is needed to determine why these characteristics impact attitudes and how to take advantage of what is being done in psychology, social work, and other academic and applied mental health, substance abuse, and social service training programs to promote the adoption and use of EPBs.

The implementation of EBPs is a particularly critical change effort that can improve the quality and outcomes of mental health services. In light of recent healthcare policy change under the Patient Protection and Affordable Care Act it is likely that the landscape of health services in the United States will also change. For example, mental health and physical health services may be more integrated in the same service settings, and federally qualified health centers (FQHCs) are likely to be implementing EBPs at an accelerated pace. Understanding the impact of organizational social context on staff readiness and willingness to implement EBPs in light of these changes can suggest ways to improve implementation. Finally, changes in healthcare policy and the financing of behavioral health, and the reorganization of the systems delivering these services (i.e., primary care and specialty mental health payment and provider networks) are likely to affect both attitudes towards EBPs (may reinforce positive attitudes or not) and organizational processes (may produce different values, may increase stress and burnout, may privilege proficient organizations) that are central to successful implementation.

A number of innovations being adopted by mental health authorities expand the definition of EBP beyond efficacious treatment models. Some examples include continuous data monitoring systems, alerts to target prescribing practices, business engineering to target high volume service users, and client outcomes monitoring and feedback to clinicians. Some of these may be considered to be evidence-based if they are proven to improve outcomes. The EBP referent for the EBPAS is more traditional psychological or psychosocial treatment that has research support and may be manualized and structured. Thus, attitudes towards therapies, clinical interventions, and treatments may be a special case of EBPs tapped by the EBPAS. Further research is needed to better understand the role of social context in attitudes toward a variety of other technical innovations as described above. Attitudes toward clinical practices and treatments may not reflect attitudes toward such innovations. The first author is currently piloting an adapted version of the EBPAS to be used to assess attitudes toward adopting specific innovations or practices.

The present study adds to the research base demonstrating that organizational social context is associated with clinician attitudes toward the adoption and use of EBPs [50, 65]. While focusing on the organization is only one aspect of the landscape of EBP implementation [1], it is an important one for a number of reasons. First, most EBPs are implemented within organizations and organizational characteristics such as social context predict provider attitudes toward adopting EBP, even when accounting for individual-level covariates. Second, organizational social context is malleable. That is, randomized controlled studies have documented that change in organizational culture and climate, as well as associated improvements in EBP implementation, can be accomplished with targeted improvement strategies [40, 50‐52, 66]. Third, we know that the availability of clinical training and funding may be a necessary but not sufficient condition for successful EBP implementation and that successful implementation strategies must address the context in which EBPs are offered [1, 4]. In conclusion, this paper adds to the growing evidence that organizational social context must be addressed in efforts to provide effective mental health treatment services through implementation of the most effective treatments. By improving the organizational social context for EBP we not only improve the work environments for clinicians, but also support the implementation of effective clinical services that decrease symptoms and improve functioning and quality of life of those who are served.