Appraising the uptake and use of recommendations for a common outcome data set for clinical trials: a case study in fall injury prevention

This study aimed to assess the impact of the ProFaNE recommendations for core outcome measures, particularly within randomised trials of fall injury prevention in older people, and found that the recommendations have had a limited impact. Although adherence to the recommendations was poor in some domains, the level of adherence was better within the domain of falls, which is the primary outcome for most trials in this field. The number of studies citing the recommendations has increased over time and by geographical region. It is likely that this trend will continue, as increased usage leads to knowledge of the recommendations becoming more widespread and improved practices in fall prevention research as a whole.

A key issue is lack of consistency in adherence to the recommendations. The recommendations in the falls domain were generally well-followed; most of the examined trials reported using the recommended methods and measures, such as the definition of a fall, daily prospective recording and summarising results by the number of falls, fallers and frequent fallers. Although it is important to note that the recommended definition of a fall and very similar definitions were in use prior to the ProFaNE initiative [6]. However, the majority of the randomised trials of fall prevention did not consider the domains of self-efficacy, quality of life or physical activity. Where these domains were reported, the level of adherence to the corresponding recommendations was poor. Although adherence varied by trial and domain, some studies followed almost all recommendations for the domains on which they reported; most notably Elley et al. reported on all five recommended domains and achieved the equal highest level of adherence of the included studies in each of the domains except the falls domain where adherence remained above average (Additional file 2) [7]. Only two trials did not report on the falls domain; Vaapio et al. only reported on health-related quality of life and Schepens et al. did not report on any of the recommended domains, instead assessing effectiveness of a fall prevention intervention by examining uptake of specific fall prevention behaviours and evaluating knowledge of fall threats [8, 9].

Only one study used the recommended measure of the mFES. However, the ProFaNE consensus acknowledges the need for further modification of this measure. Several studies used outcome measures for self-efficacy which were developed after the ProFaNE recommendations were published. Indeed, four trials used the FES-I which was developed to address the deficiencies of the mFES highlighted by the ProFaNE consensus [10].

Although only 8 of the 34 trials reported on health-related quality of life, half of these trials used a recommended measure, which is encouraging. However, further increases in the use of the recommended health-related quality of life measures are needed to facilitate health economic analysis and to capture the broader effects of fall prevention intervention, which could be quite substantial.

Around half of the randomised trials of fall prevention assessed fall-related injuries, yet none used the recommended measure or classification system of injuries, or presented the recommended summary of results. However, a possible cause for the low adherence to recommendations in the injury domain is that the corresponding systematic review was not published until 2012 [11]. In contrast, the review on the falls domain was published in 2006 and has been widely cited, thus increasing awareness [12]. Adherence to the recommendations on injury may, therefore, increase in future with the rise in supporting evidence. It is also possible that changes in injury pattern, and of access to more robust routine data means that this area of the consensus needs to be updated.

This study also found that fall prevention trials used measures of physical activity that were not considered in the original consensus, such as the Human Activity Profile Test [13]. The original ProFaNE recommendations also stimulated the development of the Phone-FITT measure, which was used by one study [14].

In addition, the increase in the number of observational studies citing the recommendations is notable; however, since the recommendations were developed for randomised controlled trials, adherence to the recommendations in non-randomised studies has not be considered in this study.

This study is one of the first to assess the impact of a recommendation for a core set of outcomes [1, 15, 16]. It includes valuable information on the types of studies that cite the recommendations, where these studies originate from, and how the citation pattern has changed over time. Several databases were searched to identify publications that cited the recommendations, and eligibility was not restricted by language.

The results are limited because impact has only been assessed using publications that cited the ProFaNE recommendations. It is possible that the true impact of the recommendations is greater than estimated. Some trials may have used the recommendations without citing the corresponding article or may have cited the corresponding systematic reviews that formed the basis of the recommendations [11, 12, 17, 18]. Likewise, adherence to the recommendations was assessed based on the information reported in each publication. Although this is a reasonable approximation, it may not reflect current practice. Trials may follow recommendations without explicitly referring to them. Moreover, trials published in the period immediately following the publication of the recommendations may not have had sufficient time to enact any changes to their methodology. A more accurate estimation of the uptake of the recommendations could be produced by conducting a systematic review of randomised trials to compare practice prior to and following publication of the recommendations, akin to Bautista-Molano et al. [15].

A further limitation is that the analysis was restricted to the earliest article published on the trial which reporting post-intervention results for the primary outcome. This could result in underestimation of the proportion of articles reporting certain domains since secondary outcomes and longer-term follow-up are sometimes reported in separate publications.

Methods for estimating the level of impact were limited due to difficulties in finding an appropriate denominator. Using the Cochrane review in the area allowed us to approximate the proportion of studies citing the ProFaNE guidance; yet this was limited as the included studies were only published up to 2012. Hence, as this study aimed to assess impact until 2015, this estimate may not be accurate due to the variation in publication rates over time. In addition, as this is one of the first studies to evaluate the impact of a core outcome set, the relative impact compared to other recommendations is unknown.

Our study suggests that little adjustment is required to the recommendations in the falls domain. Few of the examined trials reported considering the lay perspective when ascertaining information on falls. However, it seems likely that this was due to lack of reporting, not poor practice, as none of the trials reported the specific phrasing they used when questioning participants on prior falls.

Poor and inconsistent adherence to the other domains highlights that improvements can be made. The recommendations in the injury domain most urgently require improvement. The recommendations that coincide with those in the general falls domain had high levels of adherence, but those specific to fall-related injuries were not followed. Most trials considered any fracture resulting from a fall that was reported by a participant. None of the studies reported using radiological confirmation, categorising by the International Classification of Diseases or considering peripheral fractures separately. The ProFaNE recommendations can be improved by expanding the set of injuries considered beyond peripheral fractures, such as by including head injuries or vertebral fractures. It is possible that the International Classification of Diseases classification is considered too burdensome for data collection. A simplified approach to classification may encourage the presentation of different types of injuries separately, such as the system for categorisation proposed by Schwenk and colleagues based on symptoms and medical care use [11].

The findings confirm deficiencies discussed in the original recommendations. The recommended measure for assessing psychological consequences of falling via self-efficacy must be clarified. In light of this, it is planned to review and potentially update the recommendations.

The ProFaNE consensus has motivated further research into the methodology and conduct of fall prevention trials, such as the development of new outcome measures and the ProFaNE taxonomy for classification of fall prevention interventions [10, 14, 19]. Any update of the ProFaNE guidance should also involve consideration of additional instruments for measuring exposure to physical activity for inclusion in the core outcome set, such as the more recently developed Phone-FITT [14].

Overall, evaluating the impact of the ProFaNE recommendations has been a useful process. Evaluation through a review of uptake and usage can show how the core outcome set is being used in practice. It has highlighted key areas where recommendations were not being followed, indicating that future update or clarification is needed. Regular evaluation of the impact can provide evidence of deficiencies in the core outcome set to inform discussion about whether the recommendations are current and fit-for-purpose.

As stated by the COMET Initiative, there should not only be a focus on the development of core outcome sets, but also consideration to methods for dissemination. Several studies (7 of the 34 randomised trials on fall prevention in older people) included an author who was involved in the development of the ProFaNE recommendations. In addition, the proportion of studies citing the ProFaNE recommendations was much greater in regions where members of the outcome consensus group are based, estimated as 16 % in UK and Europe and 12 % in Australia and New Zealand compared to 7 % in Asia and 0 % elsewhere. Therefore, the involvement of an international and diverse group when developing a set of core outcomes measures can not only increase the applicability of recommendations but could potentially increase impact through improved reach. When creating recommendations for core outcome measures, strategies to maximise inclusivity at the time of development could be beneficial in terms of increased usage and wider dissemination. Moreover, dissemination of future recommendations for core outcome measures is likely to be greater and faster, due to increased usage and availability of modern communication technology. For example, social media can allow authors to rapidly increase awareness which could be used to improve dissemination, as well as involvement in development.