Background
The Organizational Readiness for Knowledge Translation (OR4KT) questionnaire was developed to assess healthcare organizations’ readiness to change clinical practice in order to adopt proven interventions [
1]. Implementing evidence-informed practice represents an ongoing challenge [
2]. In this paper, we focus on health promotion in primary health care, an excellent example of the huge gap between knowledge and practice. Despite the sound epidemiological evidence for the impact of individual behavior on population health [
3], the success of implementing interventions for lifestyle promotion in routine primary care has been limited to date [
4]. It is generally accepted that despite their good knowledge and positive attitudes towards health promotion, primary healthcare providers face a range of organizational and contextual barriers to providing lifestyle interventions [
5‐
8]. In this context, assessing primary care organizations’ level of readiness for lifestyle promotion interventions could help us improve the chances that these practices are successfully adopted.
Several instruments have been developed to assess organizational readiness to change [
9‐
11], but few of them specifically concern the implementation of evidence-informed health promotion interventions in primary care. For instance, the Organizational Readiness for Implementing Change (ORIC) was developed by Shea et al. [
12] drawing on Weiner’s theory of organizational readiness for change [
11]. Authors assessed its content adequacy, structural validity, reliability, and construct validity in a series of studies. Although the ORIC shows promise, more research is needed to test for convergent, discriminant, and predictive validity and assess its performance in a primary care setting [
12].
Further, the Assessment of Readiness for Chronicity in Health Care Organizations (IEMAC from the Spanish,
Instrumento de Evaluación de Modelos de Atención ante la Cronicidad) is a measurement tool specifically designed to assess readiness to implement the conceptual framework of the Chronic Care Model in the Spanish national health system [
13]. This instrument allows healthcare organizations to perform self-assessments regarding their readiness to provide integrated care for coping with chronicity and to identify areas for improvement in chronic care. Mira et al. [
14] assessed the preliminary validity and reliability of the IEMAC. Although this instrument presents fair psychometric properties, it has not been designed to assess organizational readiness to implement a specific intervention.
Weeks et al. [
15] developed the Diabetes Care Coordination Readiness Assessment (DCCRA) to assess primary care clinic readiness to coordinate care for adult patients with diabetes. This self-administered tool can be used for informing primary care organizations of their readiness for care coordination and identifying gaps in current practices. It can also be administered over time to assess progress in care coordination capacity, inform program evaluation, and assist quality improvement initiatives. Although the DCCRA could potentially be used to assess readiness for care coordination in other chronic conditions seen in primary care, it has not been adapted for wider use.
Recently, the OR4KT has been proposed as a generic instrument to assess the readiness of healthcare organizations for change to enable the implementation of evidence-informed practice in the field of chronic care [
1]. In the development of the OR4KT, organizational readiness for change was considered as a multidimensional collective construct, covering both the psychological (i.e., motivational) aspects of the members of an organization and the structural factors related to human and technical resources, as specified in the protocol [
16]. It combines generic scales adapted from existing validated ORC instruments that can be tailored to specific changes depending on the organizational context [
17]. Thus, the OR4KT operationalizes concepts for the assessment of healthcare organization capacities to engage in any kind of evidence-based change. Its main focus is on the implementation of change, but we hypothesize that the OR4KT could also be useful for monitoring change progression in an organization.
This tool was developed in English and has been translated into French and Spanish and cross-culturally adapted for use in Canada and Spain. The aim of the present study was to assess the validity and reliability of the Spanish version of the OR4KT in the context of the implementation of evidence-based interventions for health promotion and chronic disease prevention in primary healthcare organizations in the Basque country. Specifically, the OR4KT was used in the Prescribe Healthy Life III (PVS-III, an abbreviation from the Spanish:
Prescribe Vida Saludable III) project, whose objective was to optimize health promotion practice in primary care through implementation research [
5‐
7,
18,
19]. The implementation strategy used in PVS-III relies on discussion and consensus meetings for needs assessment and bottom-up decision making. Accordingly, this strategy requires the commitment of the majority of the professionals from highly motivated primary care centers. The OR4KT was used to identify primary care centers ready to initiate this practice-changing process.
Methods
Design
We conducted a cross-sectional study with repeated measures in three integrated healthcare organizations of the Basque Health Service (Osakidetza) in order to validate and assess the reliability of the Spanish version of the OR4KT questionnaire.
Development of the OR4KT questionnaire
The OR4KT was developed following a three-phase process. In the initial development phase, we first conducted systematic reviews of the theoretical underpinnings of organizational readiness (OR) and existing OR measurement instruments to provide a pool of relevant conceptual dimensions and related items [
17,
20]. Then, we conducted a modified two-round Delphi survey among international experts to assess content validity of the pilot OR4KT instrument [
21]. In the second development phase, the preliminary OR4KT instrument (containing 91 items) went through an item-reduction process, involving a panel of seven international experts, seeking to decrease its length. The final OR4KT questionnaire includes 59 items assessing 6 dimensions and 23 sub-dimensions related to the concept of organizational readiness. The third development phase was the translation of the pilot OR4KT instrument, originally developed in English, into French and Spanish and its cross-cultural adaption for use in Canada and Spain [
1].
Setting and participants
Convenience cluster sampling was carried out to select participants among the staff of 49 primary care units (Unidades de Atención Primaria—UAP) belonging to three integrated care organizations of Osakidetza. We sent a letter to the heads of these primary care units inviting their organization to participate in the project and we received a positive response from 20 of these units. Seven key informants from each unit were identified using snowball sampling, obtaining a total of 140 individual participants.
The first participant was the head of the unit, who had the official management responsibility for the unit staff, and he or she was asked to recruit one representative of each of the following groups: physicians, nurses, and administrative staff. Each of these representatives was then asked to recruit another individual among colleagues from their professional category (physician, nurse, or administrative staff) who they believed represented an average position in terms of their attitude towards health promotion, that is, neither very willing nor very reluctant.
Data collection
The survey was composed of the Spanish version of the OR4KT, a culturally adapted Spanish version of the Survey of Organizational Attributes for Primary Care (SOAPC) [
22], and characteristics of the primary care unit, as well as questions concerning the sociodemographic and professional characteristics of respondents. We also documented the number of healthcare professionals who formally engaged as collaborators in the PVS-III project within each participating primary care unit.
The Spanish version of the OR4KT instrument (Additional file
1) comprises 59 items and assesses 6 dimensions and 23 sub-dimensions related to organizational predisposition to knowledge translation: organizational climate, organizational support, contextual factors, change content, leadership, and motivation. It uses a 5-point Likert scale and the total OR4KT score is computed by summing the scores on each item, with a maximum score of 295 points. This score is then normalized on a 0 to 100 scale to ease interpretation.
We created a translated and culturally Spanish adapted version of the SOAPC, a 21-item questionnaire that assesses four organizational attributes that are relevant to primary care from the perspective of physicians, nurses, and other staff. The four scales of the SOAPC are (1) communication: healthcare professionals’ capacity to work as a team and solve problems through discussion (4 items); (2) decision making: healthcare professionals’ participation in decision making (8 items); (3) stress/chaos: healthcare professionals’ workload and the management of tension in the center (6 items); and (4) history of change in the organization (3 items). The items are rated on a 5-point Likert scale, from 1=totally disagree to 5=totally agree. The scores obtained are also transformed to obtain a normalized score ranging from 0 to 100 to facilitate the interpretation of the results. This version of the SOAPC has been validated using an expert panel and factor analysis, which confirmed its four-factor structure. The internal consistency of the scales was satisfactory, with Cronbach’s alphas of 0.81 for communication, 0.88 for decision making, 0.85 for stress/chaos, and 0.73 for history of change.
As indicated above, the survey instrument also included ad hoc questions to collect data on respondents’ sociodemographic (age, sex, and level of education) and professional (profession, position, work experience, and time in current position) characteristics. We also collected information on the primary care units, namely number of registered patients, number of healthcare professionals, mean number of registered patients per family physician or pediatrician, and a socioeconomic deprivation index for the catchment area that combines variables related to employment (unemployment rate, manual and short-term employment) and education (educational attainment rate among young people and overall).
The survey was completed a first time and then repeated under similar circumstances after an interval of 15 to 30 days.
Finally, for each participating primary care unit, we estimated the level of commitment to the project by calculating the proportion of professionals who engaged as collaborators in the PVS-III project. To do so, we held a 2-h informative meeting at each of the 20 selected primary care units to explain the objectives and methods of the PVS-III project. After the meeting, we collected individual signed consent forms from attendees who agreed to participate and collaborate in the project. Then, we calculated the percentage of professionals who committed to the project in each unit as an indicator of their intention to change their clinical practice to integrate health promotion in routine care. Units were categorized as ready to change clinical practice if more than 50% of their staff gave written informed consent to collaborating in the PVS-III project.
Data analysis
To assess the psychometric properties of the Spanish version of the OR4KT questionnaire, we used classical test theory. The reliability of the measurement was quantified in terms of consistency and repeatability. Cronbach’s α coefficients were calculated to determine the internal consistency of the scales. A value of α ≥ 0.7 was considered good. The repeatability was determined by calculating the Coefficient of Repeatability (CR) [
23].
To assess the instrument validity quantitatively, we considered internal and external evidences [
24]. Internal evidence of the relationships between elements of the test and its agreement with the underlying theoretical model was evaluated through a confirmatory factor analysis (CFA) by the maximum likelihood mean-variance (MLMV) adjusted method, which is robust to non-normal distributions. In a first step, we adjusted a model in which we specified the sub-dimension of membership for each of the items. In the next step, we carried out a second-order analysis, checking the fit of a model in which each sub-dimension was loaded in its reference dimension and these in turn in a global dimension that measures the organizational readiness. The degree of fit to the data for each model was evaluated by reference to the value of the chi-square/degree of freedom ratio and also to the incremental Comparative Fit Index (CFI) and the Standardized Root Mean Square Residual (SRMR). Models with
\( \raisebox{1ex}{${\chi}^2$}\!\left/ \!\raisebox{-1ex}{$ df$}\right. \) ≤ 5, CFI ≥ .90 and SRMR ≤ .08 are considered acceptable [
25]. As external evidences, we assessed concurrent validity by exploring the association of the OR4KT dimensions with (1) the dimensions measured by the SOAPC, which includes similar and related constructs, and hence, it was hypothesized that the relationship would be positive and (2) the proportion of professionals who individually signed the commitment to participate in PVS-III, with an expected positive relationship, as individual willingness would correlate positively with organizational readiness. To investigate these potential relationships, bivariate correlations were calculated using Spearman’s rank order correlation (rho). Finally, we analyzed the known-groups validity through receiver operating characteristic (ROC) curves constructed from the mean total OR4KT scores in the health centers in which more than 50% of the staff signed consent forms agreeing to collaborate in the PVS-III project and in those in which this requirement was not met.
We used SAS (v. 9.2, SAS Institute, Cary, NC, USA) and R (R Development Core Team, 2014) to perform all statistical analyses.
Discussion
This study aimed to provide an initial assessment of the validity and reliability of the Spanish version of the OR4KT instrument in the context of the implementation of the PVS-III project, which seeks to strengthen health promotion and chronic disease prevention in primary healthcare organizations of the Basque Health Service. Based on a collaborative approach and shared decision making between primary care providers and community members, the PVS-III model requires a high level of intra- and inter-organizational cooperation as well as the involvement of multiple sectors [
5‐
7]. Given this, it was deemed essential to assess the readiness of primary healthcare organizations to implement this approach, to enable us to carefully select centers with the highest motivation for adopting the proposed change.
The results of the initial validation process show that the Spanish OR4KT questionnaire is a reliable instrument, based on the assessment of the internal consistency and repeatability. The Cronbach’s alpha for the overall OR4KT instrument is .95, and the overall CR is 6.92%, statistics which provide strong support for the reliability of this instrument.
Compared to other available OR assessment instruments (e.g., [
8,
14,
23]), the internal consistency of the OR4KT is high, with four of the six dimensions showing Cronbach’s alphas of .80 and higher, while the two other dimensions—climate for change and contextual factors—had alpha coefficients of .79. Initial checks showed that removing some items from these two dimensions would increase these coefficients to above .80. Nevertheless, we decided to maintain all the original items from the OR4KT in the factor analysis model given that it was the first attempt to validate this instrument in a specific implementation project.
The overall repeatability score of the OR4KT was highly satisfactory and satisfactory for the six dimensions. A CR < 10% is considered to indicate excellent repeatability, and a CR between 11 and 20% is considered adequate [
26]. It is worth noting that very few OR instruments have reported repeatability to date [
17], and in this sense, our study contributes to knowledge of the reliability of OR measures.
The validity of the OR4KT was assessed by three methods, considering internal as well as external evidence. First, we performed CFAs to provide internal structure validity evidence by assessing the degree to which individual items fit the underlying construct of interest. The goal of this structure was to integrate the configuration of the items by combining six first order factors with one second order factor, the OR4KT total score. Initially, CFA on the item parcels created by the sub-dimension composites was performed. Later, a second order structure beyond the sub-dimension composites was evaluated. Both models showed acceptable fit. The loads are greater than 0.40 for most items and sub-dimensions, except for items 8, 15, 29, 32, 50, and 52 with loads between .30 and .40. Given the limited sample size and the fact that this study was conducted in a specific implementation context, it is premature to suggest changes to the OR4KT structure and measurement items at this moment. Once the metric properties of the English and French versions are analyzed, the actions to be performed with the items that have less than optimal operation will be studied.
Second, we assessed the relationships between the six dimensions of the instrument. All correlations were positive and significant.
Third, to assess concurrent validity of the OR4KT questionnaire, we explored the association between the dimensions assessed by the OR4KT and those assessed by the SOAPC, using Spearman’s correlations. Significant correlations were found between dimensions that are similar in the two instruments but not between dimensions that are unique to one of them (e.g., no correlation was found between motivation in the OR4KT and history of change in the SOAPC). The fact that several dimensions from the OR4KT and the SOAPC are correlated suggests further exploring the relative contribution of these instruments to assess healthcare organizations’ readiness to implement evidence-informed change. The SOAPC has been developed specifically for primary care organizations whereas the OR4KT could be used across a wide range of healthcare organizations.
Also, we tested the known-groups validity of the OR4KT using ROC curve analysis in order to determine the OR4KT score that best differentiated centers with a high level of commitment (i.e., those where at least 50% of the professionals signed individual consent forms agreeing to participate in the PVS-III project). The results indicate that a score of 64.48 (out of 100) on the OR4KT was the optimal cut-off point for this purpose. That is, an OR4KT score above this value would be indicative of an organization with good readiness for implementing the intervention under study.
Strengths and limitations
This study reports the first validation and application of the Spanish version of the OR4KT instrument that assesses an organization’s level of readiness to implement evidence-informed practices. The OR4KT may be useful as a screening tool to select organizations with strong potential for successful implementation of evidence-informed health promotion practices in primary care. Like other available OR instruments, such as the IEMAC [
14] and the DCCRA [
15], the OR4KT could also be used to evaluate the implementation of new practices and to follow changes over time, or as a monitoring tool for quality improvement. However, these applications have not yet been tested for the OR4KT.
This initial validation of the OR4KT questionnaire was conducted alongside the real-life implementation of the PVS-III project to improve health promotion practices in primary care. This pragmatic approach has allowed us to collect useful data to inform future implementation strategies of the PVS-III project, which constitutes a major strength of this study. However, the study has some limitations. First, the number of eligible primary care centers was limited to 49, and the final response rate was 40%, thereby providing a sample of 20 organizations. Nevertheless, the number of individual respondents from participating organizations that completed the test (127) and retest (90) surveys can be considered acceptable.
Second, the OR4KT is built as a combination of measures from several OR instruments. Given our choice to keep the OR4KT questionnaire convenient for busy healthcare providers and managers, item reduction was performed, to minimize the number of questions and hence the time required to complete the assessment. However, it is possible that the deletion of some items has modified the structure of the constructs.
In addition, the results obtained from the ROC analysis should be taken with caution for 5 reasons: (1) the sample size is too small, as so few of the centers seemed to be ready in the first place; (2) we consider that those centers in which the majority of the professionals commit to participate and sign collaboration consent are clearly different from those in which those professionals are in minority. Absolute majority is the most popular method to make group decisions and reach group consensus, which are the fundamental procedure under the process of collaborative modeling of PVS programs, but it is an experiential cut-off. (3) The scores from all facilities ranged from 52.10 to 68.94; (4) three of the facilities where less than 50% of physicians signed up had a higher score than a facility where > 50% of physicians signed up; and (5) the difference between the cut-off score and the next lowest score (which was achieved by a facility considered as “disengaged”) is small (63.45 vs. 64.48).
Future research
We consider that factorial invariance is a desirable property of a measurement instrument, so we expect to perform a progressive invariance analysis between versions (configural invariance, metric invariance, and strict factor invariance) once we have the necessary data for the analysis of the metric properties of each version of the questionnaire.
We also aim to carry out additional research to verify if the adaptation of the OR4KT items to the specific context, scope, and type of intervention to be implemented (e.g., the PVS-III program) produces changes in the metric properties of the instrument with respect to the current version of the questionnaire, in which generic items were used alongside an introduction in the instructions on the context of applying the intervention.
Finally, it would be convenient to carry out more analysis of known groups once we know that health centers of participants in PVS-III have reached the standards to consider that they have modified their preventive practice.