Discriminant content validity of a theoretical domains framework questionnaire for use in implementation research

Healthcare professionals routinely deliver pharmacological and behavior change interventions to their patients to promote health and prevent disease. However, as the evidence-base for effective interventions is continuously developing, the transfer of such evidence into routine practice often does not happen as desired[1‐3]. For example, primary care-based interventions for increasing physical activity (PA) are effective[4‐7], yet rates of PA counseling by healthcare professionals are suboptimal[8, 9], as is the fidelity of delivery of PA interventions[2, 10, 11]. This gap between research and practice reduces the impact that effective behavior change interventions can have on public health[12, 13]. Implementation research aims to bridge this gap by investigating methods to promote healthcare professionals’ uptake of research findings, including the study of factors influencing healthcare professional behavior[14, 15].

Improving the adoption and implementation of evidence-based interventions into routine practice involves changes in healthcare professionals’ behaviors that may be influenced by a range of individual, organizational, and social factors[16‐20]. Identifying the key factors associated with healthcare professional behavior can provide a basis for developing interventions to help healthcare professionals to use research findings more effectively[14]. Given the range of potential factors associated with behavior, many advocate the use of theory to guide the selection of factors to investigate[15, 21‐23]. In addition, the UK Medical Research Council guidance on developing and evaluating complex interventions recommends the use of theory in the intervention development phase[24]. The advantages of a theory-based approach are numerous: theory allows for a shared understanding, for the development of a cumulative science that limits the re-invention of existing concepts, and importantly is based on constructs which have been investigated, for which measures can be validated and standardized and have been shown to provide a useful account of behavior[25]. Furthermore, investigating the relationship between theory-based factors and healthcare professional behavior provides an opportunity to identify factors that can be targeted by implementation interventions to change healthcare professional behavior[15, 23, 26, 27].

The number and heterogeneity of potential theories that might be used to guide implementation research poses a challenge to researchers wanting to assess and identify theory-based factors underlying healthcare professional behavior[22, 28‐30]. The Theoretical Domains Framework (TDF)[31] was developed as an integrative framework of theories of behavior change to overcome these challenges. The framework includes 12 theoretical domains of potential behavioral determinants and provides exemplar questions for the theoretical assessment of implementation problems. The framework has been used in a number of studies and was demonstrated to be useful for the development of qualitative[32, 33] and quantitative[34‐36] measurement tools to assess potential implementation behavior determinants. However, factor analysis implied that only one out of these three questionnaires was able to measure the theoretical domains independently[36]. Furthermore, the questionnaires were developed to assess determinants of specific implementation behaviors in specific settings (i.e., tobacco use prevention and smoking cessation in dental healthcare[34], smoking cessation in maternal care[35], and different types of patient safety behaviors in hospitals[36]) and internal consistency reliability was low[34] or could be improved[35, 36].

Since its original development, the consensus study that produced the TDF[31] has been validated, leading to Cane et al.’s[37] refined TDF. It extends the original TDF to include the following 14 domains: Knowledge; Skills; Social/professional role and identity; Beliefs about capabilities; Optimism; Beliefs about consequences; Reinforcement; Intentions; Goals; Memory, attention and decision processes; Environmental context and resources; Social influences; Emotions; and Behavioral regulation. Main differences between the original and the revised framework include the separation of the domain Optimism from the domain Beliefs about capabilities and the domain Reinforcement from the domain Beliefs about consequences. In addition, the domain Motivation and goals was divided into two separate domains, i.e., Intentions and Goals, and the domain Nature of the behaviors was omitted in the revised framework. Although the framework is suggested to be useful for the development of theory-based questionnaires for use in implementation research, the content of the TDF has not yet been validated on item level. Therefore, it is not clear whether questionnaire items based on this recent version of the framework will be able to measure the 14 domains independently.

In the present study we aimed to develop a questionnaire assessing the 14 TDF domains, worded in such a way to provide researchers the capacity to tailor the items to the targets, actions, contexts and times of interest[38], whilst retaining the essential theoretical content in each item. Furthermore, we aimed to test the discriminant content validity of each item within the questionnaire.

Table 3 shows the results of the DCV analysis. Of 79 items, 32 were classified as measuring the intended domain and therefore included in the final questionnaire. Forty-seven items were allocated to more than one domain, of which 39 items were allocated to the intended domain as well as additional domains, while eight items were classified as measuring a domain other than the item aimed to measure. Table 4 shows Kappa values for the agreement between judges based on all 79 items of the initial questionnaire and the 32 items included in the final questionnaire. The final lists of items measuring TDF domains are shown in Table 5 (English) and Table 6 (Dutch).

We have developed a TDF-based questionnaire in both English and Dutch able to discriminately assess the majority of domains. For the first time, items have been operationalized to assess TDF domains using theoretical constructs within each domain and these items were judged to be either pure measures of the domain, or else also measuring other domains. Our findings provide an additional level of validation for the content of the TDF: not only do judges agree about the constructs within each domain and the domain structure as demonstrated by Cane et al.[37], but the majority of TDF domains have now been shown to be largely discriminately measurable. These results correspond with Taylor et al.[36, 63] who found good discriminant validity of TDF domains in a questionnaire measuring influences on patient safety behaviors[36] and in the Determinants of Physical Activity Questionnaire[63]. While Taylor et al.[36, 63] used specific items (i.e., related to a specific application), our items are generic and allow for application within a range of different contexts in which implementation research takes place. In summary, the development of our questionnaire provides important evidence of content validity and is a first step towards the development of a valid and reliable questionnaire to measure TDF-based factors underlying healthcare professionals’ specific implementation behaviors.

Of the 79 items assessed, 32 items were able to discriminately measure the following 11 domains: Knowledge, Skills, Social/professional role and identity, Beliefs about capabilities, Optimism, Beliefs about consequences, Intentions, Memory, attention and decision processes, Environmental context and resources, Social influences, and Emotion. For each of these domains at least two items were identified that can be used in the development of a TDF-based questionnaire.

Following judges allocations, items were not able to measure the domains Reinforcement, Goals, and Behavioral regulation. Items intended to measure these domains were allocated to multiple domains or classified to a domain other than the item intended to measure. This may be due to a few reasons. First, it is possible that the items used to operationalize the constructs within these domains were not appropriate, which might be related to the fact that some of Reinforcement and Goals items were newly developed by the researchers rather than previously-validated items. Nevertheless, items intended to measure the domain Behavioral regulation through the constructs Automaticity, Self-monitoring, and Action planning were adapted from previously published questionnaires, and thus it is unlikely that the existing level of validation of items is responsible for challenges in allocating items to particular domains. Second, it might be that items could not be classified to measure these three domains, because the domain definitions were not fit for purpose. This is associated with the finding that five, three, and two judges did not allocate any of the items to, respectively, the domains Reinforcement, Goals, and Behavioral regulation. The findings may also be explained by the use of domain definitions instead of construct definitions to allocate items, while items were previously developed to target individual constructs rather than broader domains. The allocation of items to domain definitions might therefore be influenced by the closeness of the definition of the domain to the definition of its constituent constructs. Finally, it could be that the remaining domains themselves cannot be discriminately measured. This seems a plausible explanation, as the domain Reinforcement is a refinement of the Beliefs about consequences domain and was originally included within the latter domain in the original TDF[31]. It is then perhaps not surprising that the Reinforcement items were judged to be assessing Beliefs about consequences, and arguably, such assignment is theoretically appropriate. Furthermore, the refinement of the domain Motivation and goals of the original TDF[31] into the domains Goals and Intentions in the recent version of the TDF and the classification of multiple goal-related constructs to the domains Goals, Intentions, and Behavioral regulation imply overlap between these domains. Therefore, it is perhaps also not surprising that the items measuring these domains were allocated to all three domains, and thus are not able to discriminately measure them. From a discriminant content validity perspective, taken together these results support keeping to the 12 original domains as a basis for the development of TDF questionnaires. When using the 12-domain framework[31] to develop a TDF-based questionnaire, items measuring the domains Behavioral regulation and Nature of the behaviors should be identified to maintain the comprehensive nature of the TDF. This could be done by selecting domains’ related key constructs as provided by Michie et al.[31] and selecting items from existing validated scales.

Lastly, the findings indicate that further refinement of the final questionnaire is required. In general, the amount of items measuring most of the domains could be increased to at least three items for each domain (at least three items with a loading above .80 will give a reliable component[64]). With regard to the specific domains, the final items measuring the domain Environmental context and resources are framed entirely in terms of the socio-political context, while there may be additional environmental and resources influences that remain unmeasured. The initial version of the questionnaire included items related to characteristics of the innovation, organization, socio-political context, and innovation strategies[16‐20], however, only the items assessing the socio-political context were judged to discriminately assess this domain. Lack of discriminant content validity of items measuring characteristics of the innovation, organization, and innovation strategies might be due to our method of developing a generic questionnaire based on factors related to a specific implementation behavior (i.e., the implementation of PA interventions). Moreover, the domain Environmental context and resources is arguably among the least well conceptualized domains of the TDF, which may partly explain challenges that judges faced in allocating items to this domain. Nevertheless, potential users of the final questionnaire may wish to incorporate additional more contextually sensitive items focusing on the environment and resources whilst recognizing that their discriminant content validity has not yet been demonstrated. In the initial questionnaire, items measuring the domain Emotion were adapted from previously published questionnaires. Specifically, items measuring the construct Affect were based on the Positive and Negative Affect Schedule[49] and Stress items were based on the General Health Questionnaire[48]. Items measuring the construct Stress demonstrated to be able to discriminately assess the domain Emotions, while Affect items did not. Therefore, the final questionnaire includes items concerning healthcare professionals’ general feelings (i.e., Stress) instead of their emotions related to performing a specific behavior (i.e., Affect). Yet, when investigating determinants of healthcare professionals’ implementation behaviors, items assessing emotions in relation to performing a specific behavior should also be taken into account as these have been found to be linked to implementation behaviors in previous research[65‐67]. Although initial TACT-specific items assessing the construct Affect were not judged to discriminately assess the domain Emotions, potential users of the final questionnaire may want to consider using such items by including other emotions such as pride, empathy[67], fear[65‐67], and embarrassment[66]. Furthermore, the assessment of the domain Knowledge could be improved by adding items to test healthcare professionals’ knowledge on a certain implementation behavior[66, 68].

To our knowledge, this study is the first to develop a generic (i.e., formulation of items following the 'TACT principle’[38]) TDF-based questionnaire in both English and Dutch including items which are able to discriminately measure a majority of the domains. The results partly support Cane et al.’s validation of the TDF[37] and suggest that the 12-domain version[31] might be more applicable in developing a TDF-based questionnaire. The items of this questionnaire can be used for the development of a questionnaire to measure TDF-based determinants of healthcare professionals’ specific implementation behaviors. Future research should investigate the concurrent and predictive validity and reliability of such a questionnaire in practice, among a large healthcare professional sample.

In general, a valid TDF-based questionnaire will increase the use of theory in the assessment of barriers and facilitators for implementation problems[31, 69, 70], which can inform the selection of possible techniques that can be used to change healthcare professionals’ behaviors[15, 23, 26]. Consequently, research on the development of a generic TDF questionnaire will improve our understanding of factors influencing healthcare professionals’ implementation and advance theory and methods in implementation research.