Measuring determinants of implementation behavior: psychometric properties of a questionnaire based on the theoretical domains framework

Much research and funding is invested into developing, piloting, and evaluating evidence-based innovations to promote health. However, the transfer of effective innovations, such as pharmacological and behavior change interventions, into routine healthcare practice often does not happen as desired [1‐5]. With the public health impact of these innovations depending on their implementation in practice, it is important to understand healthcare professionals’ (HCP) implementation behaviors and factors associated with suboptimal use of research evidence [6, 7].

Many factors can potentially influence HCPs’ implementation behaviors. These factors may be related to characteristics of the innovation (e.g., compatibility, complexity), social setting (e.g., norms, support), organizational context (e.g., capacity, resources), innovation strategies (e.g., training, reimbursement), patient (e.g., attitudes, compliance), and the individual HCP (e.g., skills, attitudes) [6, 8‐13]. Identifying the key factors associated with HCP implementation behavior can inform the development of strategies to promote evidence-based behavior [6, 14‐19].

Research has shown that active implementation strategies, such as educational outreach and reminders, can be effective in enhancing implementation behaviors [20, 21]. However, due to the scarce use of theory to inform the choice and design of implementation strategies [22], there is a lack of understanding of why strategies are effective or not [23]. To enhance the effective development of implementation strategies, therefore, many advocate using a theoretical approach to guide the investigation of implementation determinants [14, 17, 23‐25].

Behavior change theories provide testable hypotheses about when and why specific factors will lead to a certain implementation behavior. However, a limitation in the use of these theories to asses and identify factors underlying HCP implementation behavior is the large number of theories that might be used and their overlapping constructs [12, 25‐27]. The Theoretical Domains Framework (TDF) [28, 29] is an integrative framework that can be used to overcome this constraint. Within the original TDF [28], constructs from 33 behavior change theories were grouped into 12 domains of behavioral determinants covering the full range of current scientific explanations for human behavior (i.e., ‘Knowledge’, ‘Skills’, ‘Social/professional role and identity’, ‘Beliefs about capabilities’, ‘Beliefs about consequences’, ‘Memory, attention and decision processes’, ‘Environmental context and resources’, ‘Social influences’, ‘Emotion’, ‘Behavioral regulation’, and ‘Nature of the behaviors’). As a consequence, researchers can use this integrative framework instead of having to choose between different theories.

The TDF has instigated a new line of investigation and has been applied in many implementation studies. Specifically, qualitative studies concluded that the TDF was useful for the comprehensive exploration of possible explanations for suboptimal implementation behavior (e.g., [30‐35]) and for the identification of suitable theories to further investigate these behaviors [27, 36]. Furthermore, the framework was used for the development of questionnaires to assess potential implementation behavior determinants [37‐39]. So far, however, questionnaires’ internal consistency reliability was insufficient [37‐39], and only one out of three questionnaires was able to measure the theoretical domains independently [39]. Consequently, there is need for a valid and reliable method to identify theory-based factors influencing HCPs’ implementation behaviors to be able to design effective implementation strategies [12].

Recently, the TDF [28] has been validated, leading to the revised TDF including 14 domains [29]. Main differences between the original and the revised framework include the separation of the domain ‘Optimism’ from the domain ‘Beliefs about capabilities’ and the domain ‘Reinforcement’ from the domain ‘Beliefs about consequences’. Moreover, the domain ‘Motivation and goals’ was divided into two separate domains, i.e., ‘Intentions’ and ‘Goals’, and the domain ‘Nature of the behaviors’ was omitted in the revised framework. As a first step in the development of a TDF-based questionnaire for the valid and reliable assessment of factors influencing HCP implementation behavior, we developed a generic questionnaire assessing the 14 domains of behavioral determinants of the revised TDF [29]. Investigation of questionnaire items’ discriminant content validity based on judgments of a sample of experts on behavior change theory resulted in a questionnaire able to assess all domains discriminately, except for the domains ‘Reinforcement’, ‘Goals’, and ‘Behavioral regulation’. Accordingly, the findings suggested that the 12-domain original version of the TDF [28] might be more applicable in developing a TDF-based questionnaire [40].

The main aim of the current study was to develop a questionnaire based on the 12-domain version of the TDF [28] and to test the psychometric properties of this questionnaire on a sample of HCPs. To validate the Determinants of Implementation Behavior Questionnaire (DIBQ) the following research questions were addressed: 1) does confirmatory factor analysis support the predefined structure of the TDF-based questionnaire (i.e., construct validity); 2) is the questionnaire able to measure TDF domains in a reliable way (i.e., internal consistency reliability); and 3) are the domains of the questionnaire independently measurable (i.e., discriminant validity)? Our specific interest is in HCPs’ implementation of physical activity (PA) interventions, which we used in this study as a field of application for the DIBQ.

We developed and tested a questionnaire assessing factors influencing HCPs’ implementation behaviors that was based on a theoretical framework of behavioral determinants [28]. The DIBQ was one of the first TDF-based questionnaires that was developed in a rigorous manner, and showing very good psychometric properties. That is, it had good construct validity, and the majority of domains showed high internal consistency reliability and discriminant validity. While our focus was on the measurement of factors influencing the implementation of PA interventions in PHC, we suggest that the DIBQ can be applied more broadly, as the questionnaire can easily be adapted to other contexts in which implementation research takes place. Consequently, the DIBQ can solve previously reported problems with the measurement of theory-based factors underlying HCP behavior [12, 25‐27]. This can contribute to the development of effective implementation strategies and subsequently the impact of evidence-based interventions.

With regard to the questionnaire’s construct validity, our findings supported the majority of the predefined structure of the questionnaire that was based on the 12 domains of the TDF [28]. They correspond with Taylor et al. [39, 68], who found good discriminant validity of TDF domains in a questionnaire measuring influences on patient safety behaviors [39] and in the Determinants of Physical Activity Questionnaire [68]. These results provide an additional level of validation for the content of the TDF, and they confirm the viability of using the framework for construction of a theory-based questionnaire. Nevertheless, the questionnaire’s construct validity could be enhanced by some adjustments in content of the domains and the structure of the questionnaire to 18 domains.

The main adjustment we made to the structure of the questionnaire was dividing the domain ‘Environmental context and resources’ into five different environment-related domains: ‘Innovation’, ‘Socio-political context’, ‘Organization’, ‘Patient’, and ‘Innovation strategy’. This adjustment is consistent with leading theoretical models on the introduction of innovations in healthcare [6, 8‐12]. Replication of this domain-structure in future research may suggest including five different environment-related domains in the TDF. Next, ‘Optimism’ items were separated from the domain ‘Beliefs about capabilities’. This separation makes sense because ‘Optimism’ items were measured as a general disposition (e.g., ‘In my work as a physical therapist, in uncertain times, I usually expect the best’), whereas ‘Beliefs about capabilities’ items concerned capabilities that are required to achieve a specific outcome (e.g., ‘I am confident that I can deliver [PA intervention] following the guidelines’). Furthermore, the adjustment corresponds with the results of the recent validation of the TDF [29]. Items measuring social support from the management were assigned to the domain ‘Organization’ and ‘Priority’ items were separated from ‘Intention’ items. The first adjustment could also be justified by conceptual links between items and domains, and the latter adjustment corresponded with results of the validated TDF [29]. In addition, dividing the domain ‘Emotion’ into the domains ‘Positive emotions’ and ‘Negative emotions’ could be explained by previous research that indicated that positive and negative affect are two relatively independent constructs that can be measured discriminately [69, 70]. Based on similarities in their content, items measuring the domain ‘Memory, attention, and decision processes’ and Automaticity items were merged into the domain ‘Nature of the Behaviors’. Moreover, the link between automatic behaviors and memory was highlighted by Wood and Neal [71]. When developing a TDF-based questionnaire, it is possible that adding questions on attention and decision making to the memory items might decrease the overlap between the domains ‘Memory, attention, and decision processes’ and ‘Nature of the Behaviors’. Finally, some items measuring the domains ‘Priority’, ‘Innovation’, ‘Organization’, ‘Socio-political context’, and ‘Patient’ were deleted based on the domains’ Cronbach’s alpha values. An explanation based on the content of these items could not be found; however, lack of internal consistency reliability of the domains ‘Priority’, ‘Innovation’, ‘Organization’, ‘Socio-political context’, and ‘Patient’ might be related to the fact that the items measuring these domains were all newly developed. This suggests that items measuring the domain ‘Environmental context and resources’ can be improved (see Chaudoir et al. [12] for an overview of measures assessing these domains related to the environment).

No adjustments were needed for five out of the 12 domains of the initial questionnaire: ‘Knowledge’, ‘Skills’, ‘Social/professional role and identity’, ‘Beliefs about consequences’, and ‘Behavioral regulation’. This might be explained by the use of previously published questionnaires for the development of ‘Knowledge’ and ‘Behavioral regulation’ items, and most of the ‘Beliefs about consequences’ items. Furthermore, items measuring the domains ‘Skills’ and ‘Social/professional role and identity’ were validated by the discriminant content validity study [40]. Noticeably, the ‘Knowledge’ item ‘I know how to…,’ ‘Reinforcement’ items, and items measuring the construct Action Planning performed well, while they could not be validated by the discriminant content validity study [40]. This might be explained by the divergence in the main aims of the two studies; the increased focus on differences between individual items when investigating items’ discriminant content validity, and the emphasis on similarities between groups of items when examining a questionnaire’s construct validity. Indeed, in the present study, items that were not validated in the discriminant content validity study were surrounded by other previously validated items.

Compared to three other studies using a TDF-based questionnaire to identify implementation behavior determinants [37‐39], our questionnaire demonstrated high internal consistency reliability for the majority of domains. Explanations for this might be the lower number of items that the previous studies used to measure each domain [37‐39] and the development of items for domains instead of constructs within domains [38, 39]. Furthermore, it is not clear to what extent Beenstock et al. [38] and Taylor et al. [39] used items from previously published questionnaires.

Although OMG analyses revealed sufficient discriminant validity on item level, attenuation-corrected correlations revealed overlap between the domains ‘Knowledge’ and ‘Skills’ and ‘Skills’ and ‘Social/professional role and identity’. On the other hand, bootstrapped 95% confidence intervals around correlations suggested that the questionnaire was able to measure TDF domains discriminately. Based on these results and the different content of the domains, we did not merge them into one single domain. However, high correlations between domains might be problematic when analyzing associations between domains and outcome variables taking a multivariate approach.

While our focus was on the measurement of factors influencing HCPs’ implementation of PA interventions, the questionnaire was designed to be easily adaptable so it can be used in studies investigating implementation behaviors performed by other HCPs in other settings. However, depending on the behavior, the implementing HCP, and the context, it may be necessary to include items for specific barriers and facilitators. For example, time, patient motivation, and financial support may play a role in the delivery of PA interventions by physical therapists, while these factors might not relate to other behaviors, HCPs and settings. Moreover, validity and reliability of use of the questionnaire for other behaviors, HCPs and settings needs further investigation.

Some limitations of this study need to be taken into consideration when interpreting the results. First, respondents were physical therapists delivering PA interventions to a variety of target groups. In this study, we did not distinguish between the different PA interventions. Our results suggest sufficient internal validity of the DIBQ. However, a question remains as to whether the structure of the DIBQ holds for every specific PA intervention. In this study, small sample sizes within each PA intervention (sample sizes varied from 4 to 101) hindered the performance of confirmatory factor analysis for each PA intervention separately. A recommendation for future applications of the DIBQ is to replicate the reliability analysis for the target group at hand. Second, the questionnaire assessed TDF domains through their related constructs. However, to develop a questionnaire that is of an acceptable length to fill in, only a selection of constructs could be measured. Although the selection of key-constructs was based on previous research on factors influencing the implementation of PA interventions in primary healthcare [13, 43], it could be that some of the domains’ key-constructs are not part of the questionnaire leading to decreased validity of the measurement of domains. For example, the construct Intrinsic motivation [72] was not included to measure the domain ‘Motivation and goals’ and the construct Burnout [73] was not included to measure the domain ‘Emotion’, although we know from previous research that these are important determinants for HCPs’ evidence-based practice [74, 75]. Nevertheless, a questionnaire including 93 items might still be too long to fill in. This could also be an explanation for the 55.2% response rate, which was comparable to previous reported response rates of 54% [76] and 57% [77] in surveys among physical therapists, but can be considered low in comparison to Barrett et al. [78], who reached a response rate of 88%. A next step in the development process could be to develop a shorter version of the DIBQ and assess its psychometric properties. One strategy to decrease the amount of items would be to select items measuring the domains directly, instead of through their related key construct. Taking into account the criterion for a reliable component (i.e., at least three items with a loading above .80 [79]), this could decrease the average of 4 items for each construct to 4 items for each domain. The results of the discriminant content validity study [40] may guide the selection of items in order to obtain a shortened version of the questionnaire. Comparisons between respondents and non-respondents indicated that the latter were significantly older and had more practice experience, which limits the generalizability of our results. Finally, the methods used to validate our questionnaire were limited to factor analyses and the examination of discriminant validity of the domains, and only internal consistency reliability was assessed. Future research should also investigate items’ predictive validity and test-retest reliability of the questionnaire.

This study describes the development and initial validation of the DIBQ. The questionnaire showed good construct validity (i.e., research question 1) and the majority of domains showed high internal consistency reliability (i.e., research question 2) and discriminant validity (i.e., research question 3). Therefore, the questionnaire is viable to measure potential determinants of implementation behavior in a theory-based and comprehensive way. The identification of factors influencing implementation behaviors provides important information on what factors should be targeted when designing strategies to promote the effective implementation of interventions [6, 14‐19]. This is highly likely to increase the impact of health behavior change interventions. Future studies on the psychometric properties of the questionnaire are warranted and should go beyond construct validity, internal consistency reliability, and discriminant validity. In addition, more research is needed to understand the strengths and limitations of the questionnaire when it is used for other behaviors among other HCPs and in other settings.

In our study, completion of the questionnaire indicated participants’ consent for their participation in the study.