Understanding effects in reviews of implementation interventions using the Theoretical Domains Framework

The post-fracture management of patients at risk of osteoporosis is an area of patient care in which there is evidence of a large quality of care gap [21‐30] despite widely available evidence-based guidelines [31‐35]. Two key post-fracture management behaviours for which a quality gap has been widely documented include primary and secondary healthcare professionals scanning bone mineral density and prescribing anti-resorptive therapy.

We conducted a systematic review of interventions (reported elsewhere) [36] aiming to improve the investigation and management of osteoporosis in patients following fragility fracture. For both of the main outcomes of the review (bone mineral density (BMD) scanning and osteoporosis treatment with anti-resorptive therapy), all nine identified studies [37‐45] reported a positive effect of the intervention, with an overall 36 % absolute increase in BMD scanning rates and a 20 % absolute increase in treatment rates.

Whilst the review provided a compelling case that implementation interventions can be effective in improving scanning and treatment rates, replicability of these interventions is undermined by a lack of capacity to understand how the intervention had its effect, i.e. what factors did the interventions target to promote change.

The aim of this paper was to explore, using the TDF, the factors that the interventions may have been targeting, how this might be related to the main outcomes of the systematic review (rates of BMD scanning and osteoporosis treatment with bisphosphonate medication) and highlighting opportunities for targeting factors that seem to be less frequently considered but may be worth exploring in future interventions.

As part of the systematic review, a verbatim description of the interventions as delivered and the care received by the control group was extracted and reported. The authors of the studies included in the systematic review were contacted by email and informed of our intention to carry out a secondary analysis of their interventions. It was requested in the email that firstly they verify whether or not they agreed with the details of their intervention which were identified and described in the email and secondly send us any additional materials that were given to those delivering the intervention or any protocols or linked studies that described the intervention, how it was carried out and what it was targeting, in more detail. Those nominated as the corresponding author were contacted initially, but if the email address was no longer functioning, alternative authors were tried. A first reminder was sent 2 weeks after the initial email, a second reminder 2 weeks after that, and the authors were then given a further 2 weeks to respond.

We used the 14 domains from TDFv2 [14] as a basis for coding TDF domains. The TDF domains that appeared to be targeted by the interventions and within the control groups were identified and coded independently by two reviewers (EAL, a clinician and JP, a behavioural scientist), using a data extraction form designed for the purpose (Additional file 1). We used domains as well as constructs within domains to inform coding decisions within domains, using construct definitions as described by Cane et al. [14]. The data extraction form was tested on one included study. The coding of each domain was supported by evidence from the text. However, this required inferences to be made about which domains the authors were intending to target, as this was never explicitly stated in the text. The descriptions of the interventions were studied and each aspect that was judged to be targeting a domain with respect to the behaviours of BMD scanning and osteoporosis treatment with anti-resorptive medication was coded. The domains were also coded according to the individual targeted, i.e. patient or healthcare professional. Inter-rater reliability was calculated prior to resolving discrepancies (Cohen’s kappa: Κ = 0.507 (moderate agreement [46]). Following discussion with a third party (MPE), 100 % agreement was achieved.

The relationship between the total number of domains coded and effect size of the intervention was explored using Pearson correlations (two-tailed) for both BMD scanning and osteoporosis treatment with anti-resorptive therapy. “Total number of domains” is a sum of every time any domain was coded and does not take into account the recipient of the intervention. For example, if “knowledge” was coded twice for the primary care physician (PCP) and once for the patient, and “beliefs about consequences” was coded three times for the PCP, this would add up to a total of six times. The total number of times the domains were coded in the control groups of the studies was subtracted from the total number of times the domains were coded in the intervention groups. This was a subtraction of total number of domains and did not take into consideration which domains were coded in each group. To investigate whether subtracting domains that appear in the control group impacted on the findings, we conducted a sensitivity analysis whereby all domains coded in the intervention group were counted irrespective of whether also coded in the control group.

With regard to the development of the intervention, only two studies reported consulting with representatives of the intended professional recipients [40, 43]. Six studies reported the evidence base for the intervention [37, 38, 40‐43]. Patient involvement was not reported by any of the studies. In four studies, the authors reported specific barriers to change that the intervention was tailored to address [37, 40, 42, 44]. Only one study reported that they had carried out substantial exploratory work to identify them [40] (literature reviews and qualitative in-depth interviews with healthcare professionals). In two of the four studies, barriers were not identified and discussed until the end of the paper [37, 44]. Only one study designed the intervention to address all of the identified barriers [40]; the others offered no explanation as to why they had chosen particular barriers to target over others, with one study selecting a single barrier from a long list of barriers [44].

The coding for the domains targeted in the intervention and control groups for each of the studies is shown in Table 1. A more detailed description of the coding as well as the care given to intervention and control groups by study is presented in Additional file 2.

Table 2 presents the number of times each of the domains were coded with respect to BMD scanning and osteoporosis treatment with anti-resorptive therapy in the intervention group of each of the studies. The recipient of the intervention, i.e. patient or PCP, was also specified.

The domain coded most frequently was “memory, attention and decision processes” (10 of 10 interventions; BMD scanning coded 33 times; treatment coded 29 times). The second most frequently coded domain was “knowledge” (10 of 10 interventions; BMD scanning coded 21 times; treatment coded 20 times), closely followed by “environmental context and resources” (9 of 10 interventions; BMD scanning coded 16 times; treatment coded 13 times), “social influences” (9 of 10 interventions; BMD scanning coded 11 times; treatment coded 11 times) and then “beliefs about consequences” (5 of 10 interventions; BMD scanning coded 9 times; treatment coded 9 times). In all of the studies, there was a combination of at least four of these five domains. There were five domains that were never coded: “social/professional role and identity”; “optimism”; “reinforcement”; “intentions”; and “behavioural regulation”. The remaining four domains (“skills”; “beliefs about capabilities”; “goals”; “emotion”) were coded only once or twice with respect to both BMD scanning and treatment.

Table 3 presents the number of times each of the domains were coded with respect to BMD scanning and osteoporosis treatment with anti-resorptive therapy in the control groups of each of the studies.

We identified and coded domains in the control group for only five of the studies. This was due to the poor descriptions of the care given to the control groups. Fewer elements were coded, and these were spread over fewer domains. The domains most frequently identified in the description of the control group were again “memory, attention and decision processes” (4 of 9 control groups; BMD scanning coded 7 times; treatment coded 7 times), “knowledge” (4 of 9 control groups; BMD scanning coded 4 times; treatment coded 4 times) and “social influences” (3 of 9 control groups; BMD scanning coded 4 times; treatment coded 4 times). “Beliefs about consequences” and “environmental context and resources” were coded once each for both BMD scanning and treatment.

Across the studies described as having been tailored to identified barriers and facilitators, each reported lack of clarity regarding which physician was responsible for the investigation and management of osteoporosis following a fragility fracture as a significant barrier but the interventions themselves were not described to target the “social/professional role and identity” domain for any of these studies. The interventions instead addressed this barrier either by taking the behaviour out of the hands of the PCP altogether (“environmental context and resources”) or by using reminders for the PCP to perform the behaviour (“memory, attention and decision processes”).

On the whole, the remainder of the barriers and facilitators that were identified in the studies were reflected in the frequency of the domains coded, with the exception of the “social influences” domain. This was the fourth most frequently coded domain and yet was only referred to in one study as being important in changing health care professionals’ behaviour [41]. Only one study addressed the “social influences” domain, by selecting local opinion leaders that had been identified as educationally influential physician peers in the area of osteoporosis by PCPs through a validated questionnaire prior to the intervention [43].

Table 4 summarises the total number of times the domains were coded, for each study intervention and control group by BMD scanning and osteoporosis treatment with anti-resorptive therapy. Scatterplots are presented in Fig. 1.

There was a statistically significant inverse relationship between the total number of times the domains were coded within an intervention and the post-intervention risk difference for BMD scanning, r = −0.831, p < 0.05 but not for treatment (r = −0.615, p = 0.058). The sensitivity analysis (not subtracting the number of control group domains) showed similar results (BMD scanning: r = −0.704, p < 0.05; treatment: r = −0.603, p = 0.065).

Table 5 summarises the number of different domains coded in the intervention and control groups of each of the studies. Scatterplots are presented in Fig. 2.

There was a statistically significant inverse relationship between number of different domains coded within an intervention and the post-intervention risk difference for BMD scanning (r = −0.848, p < 0.05) but not for treatment (r = −0.530, p = 0.115). The sensitivity analysis (no subtraction of control group domains) showed similar results for BMD scanning (r = −0.728, p < 0.05), but for treatment, the relationship became statistically significant (r = −0.644, p < 0.05).

“Memory, attention and decision processes” was by far the most frequently coded domain. This suggests that the study authors believed that PCPs need reminders, prompts or decision aids to enable them to perform the behaviours, indicating that they believed that PCPs were either forgetting to perform the behaviour or had impaired decision processes with regard to performing the behaviour. This also addressed the identified barrier of PCPs failing to make the connection between a fracture and osteoporosis.

“Knowledge” was the second most frequently coded domain. For all the interventions that included the patient as a recipient, the “knowledge” domain was coded with respect to the patient. This indicates that the authors believed that in order to ensure that the patient could influence the PCP’s behaviour, they first required a certain amount of knowledge about the investigation and management of osteoporosis following fragility fracture. The use of “knowledge” with regards to the PCP was less consistent (five out of ten interventions) which suggests that the study authors differed in their views about PCPs’ existing knowledge of the condition and its management. This domain encompasses the barriers of a lack of awareness by patients and physicians of the treatment guidelines and efficacy of medications for osteoporosis following fragility fracture.

“Environmental context and resources” was the third most frequently coded domain. It was interesting that in the three studies with the greatest effect sizes for BMD scanning (Miki [44], Rozental [45] and Majumdar [40], with post-intervention risk differences of 71, 62 and 51 %, respectively), a large part of the intervention was coded within this domain. In the Miki study, the BMD scan was ordered by the orthopaedic surgeon and carried out whilst the patient was still in hospital. In Rozental the BMD scan was ordered by the orthopaedic surgeon during the first outpatient clinic visit; and in Majumdar, a case manager arranged the BMD scan. This was coded as a resource, but one could argue that rather than changing the behaviour of the PCP, the effectiveness of the interventions stemmed from the fact that the behaviour was taken completely out of the hands of the PCP and was actually a service delivery change—someone else performed the behaviour instead. In these three examples, the barrier of inadequate access to BMD scanning was addressed. It is worth noting that this domain was also coded for studies with smaller effect sizes for BMD scanning, including Solomon which had the smallest effect size (4 %) [41], and in which it was coded once for the patient, four times for the PCP and once for the pharmacist. This highlights a key point when using the TDF: it is not just which domains you target but how they are targeted that is important. Just because one targets “environmental context and resources” or “Memory, attention and decision processes’, it does not mean that it is done appropriately or effectively. However, the advantage of using the TDF is that it allows us to use a common language for describing those targets.

Finally, “Beliefs about consequences” was coded fifth most frequently. Interestingly, patient “beliefs about consequences” was targeted in all of the five studies for which this domain was coded but PCP “beliefs about consequences” was only targeted in two of them. This implies that the authors believed that patients had not considered or did not understand the consequences of failing to investigate for and treat osteoporosis post-fracture, and by remedying this, they would be able to change the behaviour of the PCP by, for example, using patient prompts. This addressed barriers such as concerns about whether bisphosphonate treatment might impair fracture-healing, concerns about adverse effects of medications or lack of awareness of the efficacy of medications following fracture.

In addition to the poor reporting of the rationale for the interventions, from some of the descriptions given, it was difficult to extract sufficient detail to be confident that the interventions were being described in a way that would make them replicable. It was also difficult to disentangle what the investigators felt was the content of their intervention (the active ingredients; cf. [47]) from the method that they chose to deliver it (e.g. printed educational materials). Such distinction is important in order to promote greater clarity in the description of interventions [48].

The descriptions of the control groups were often underreported, and in some cases completely absent. This impacted on our ability to code certain domains with confidence. This is a consistent finding throughout the behaviour change literature. Reviews of nearly 1000 behaviour change outcome studies [49‐52] found that detailed descriptions of interventions were present in only 5 to 30 %. Failing to report the actual behaviour targeted for change by the intervention or describing the content of the intervention in sufficient detail makes it impossible to identify why an intervention did or did not work [53, 54]. This prevents the replication of successful interventions in wider settings; the introduction of the Template for Intervention Description and Replication checklist (TIDieR) [55] may improve the current situation.

The exploratory analysis showed a statistically significant inverse relationship between the total number of times the domains were coded and the number of different domains coded and the effect size of the intervention for BMD scanning. However, the results for the equivalent analysis on anti-resorptive therapy were not statistically significant. This may be due to the fact that anti-resorptive therapy is itself partially dependent upon BMD scanning. We have shown that the study authors rarely documented the management guideline they were using, but in the majority of cases, the patient needed to have a BMD scan before treatment was commenced. Following the BMD scan, the patient may not have required treatment for osteoporosis so although the patient would have received appropriate care, it would appear as if the intervention had not been successful as the patient was not given subsequent anti-resorptive treatment.

Nevertheless, the inverse relationship demonstrated for BMD scanning was unexpected. We assumed that multifaceted interventions would prove to be associated with larger effect sizes. There may be a range of reasons that explain the inverse relationship, including that it is a chance finding given the small number of studies included in the analysis. Equally, it may be that targeting more domains may not be inherently better; what is likely more important may be to better match the interventions to the domains shown to be relevant for the behaviour, context and population under study [16].

None of the studies employed the explicit use of theory, which meant that our coding was based on inference from the text. This is not unusual; studies in this area rarely employ theories of behaviour change, or if they do, they fail to report it [56].

There are also challenges to using the TDF itself in this context. Not all domains are necessarily mutually exclusive, with some sharing certain constructs, for example, “action planning” is a part of both “goals” and “behavioural regulation”. Separating “intentions” and “goals” domains was sometimes a challenge. Given the inferential nature of the coding, there were instances when agreeing upon a target domain proved challenging (reflected in our inter-rater reliability). This was in line with previous research in the context of interview studies [57], perhaps reflecting the difficulty of clarifying the boundaries between some domains when using the TDF as a coding framework.

A final challenge involved coding interventions that included a service delivery change. In three studies, the behaviour of performing the BMD scan was taken out of the hands of the target PCP altogether and performed by another individual. We coded such instances as targeting “environmental context and resources” domain. However, the target of this part of the intervention was at an organisational level rather than an individual PCP or patient level. Nevertheless, someone’s behaviour higher in the organisation needed to change for this service delivery to take effect; the TDF could potentially be applied to describe their behaviour as well in future studies if descriptions in intervention reports provide such detail.