Evaluating “payback” on biomedical research from papers cited in clinical guidelines: applied bibliometric study

Data sources

Fifteen sets of guidelines on disease management were selected as data sources for the study (table 1).4^–18 The guidelines covered a range of conditions seen in general (family) practice or in hospital care, and all had been produced in the United Kingdom, either by the royal colleges or by the North of England Evidence Based Guidelines Development Project. The guidelines were selected because they had been assessed by the NHS Appraisal Centre for Clinical Guidelines. The role of the centre is to advise the NHS Executive about the quality of clinical guidelines that have been funded through the national guideline programme.19 Its appraisal consists of a structured peer review based on a validated appraisal instrument that assesses key elements for the development and reporting of clinical guidelines.20

The “knowledge cycle time”

In patent analysis, an important characteristic is the “prior art cycle time,” which is defined as the median age of the patents referenced on the front page of a patent.22 With guidelines, an analogous indicator would be the “knowledge cycle time”—that is, the age of papers cited in clinical guidelines, as illustrated in figure 1. The median knowledge cycle time for all 15 guidelines was eight years; 25% of the papers, however, were more than 10 years old and 4% more than 25 years old.

Fig 1.

Age of papers cited in clinical guidelines

• Download figure
• Open in new tab
• Download powerpoint

Country of authorship

Figure 2 shows the distribution of the addresses of authors cited in the clinical guidelines for the Group of Seven (G7) largest economies. Comparison data for outputs in 1998 were taken from the Science Citation Index with a title keyword and specialist journal search strategy developed for biomedicine and described elsewhere.1 In total 9007 authors were cited in the 1761 papers, which is an average of 5.1 authors per paper. This distribution is heavily skewed as five papers had more than 100authors. Most papers were published by authors living in either the United States (36%) or the United Kingdom (25%), with the five remaining countries accounting for 19% of addresses. Non-G7 countries accounted for 26% of the papers. Note that for a number of the guidelines the total percentages add up to more than 100% because of collaboration between the G7 countries. The most salient observation to be made from figure 2 is the higher proportion of UK papers cited in guidelines compared with biomedicine as a whole.

Fig 2.

Nationality of authors of papers cited in clinical guidelines

• Download figure
• Open in new tab
• Download powerpoint

Type of cited research

The third analysis was based on the journals in which the cited papers appeared and was used to determine the extent to which basic or clinical research is cited in clinical guidelines. Figure 3 shows the distribution of cited papers across four different journal levels. Of the 1761 papers, 260 (15%) did not have a research level. Of the remaining papers, nearly three quarters appeared in level 1 (clinical observation) or level 2(clinical mix) research journals compared with only four papers (or 0.2%) in level 4 (basic) journals.

Fig 3.

Research level of journals cited in clinical guidelines

• Download figure
• Open in new tab
• Download powerpoint

Comparison data were taken from the research output database for UK biomedical publications between 1988 and 1995 (the right hand column for each research level in figure 3). To make this comparison valid, however, the guideline papers have been restricted to the 291 UK publications between 1988 and 1995 (the middle column for each research level). As is apparent from figure 3 clinical guidelines are citing clinical papers. In itself this observation is not surprising, though there is an underlying assumption in biomedical science policy that basic research feeds into clinical practice. To see if this assumption holds, figure 4 examines the type of research cited in subsequent “generations” of papers, where a second generation paper refers to papers cited in a 5% random sample of papers cited in the guideline papers and a third generation paper to a 5% random sample of papers cited in a 5% random sample of papers cited in the guideline papers and so on. In this analysis the proportion of clinical observation (level 1) papers fell from 30% to 14% over the four generations analysed while basic (level 4) papers increased from 0.2% to 8%.

Fig 4.

Research level of journals cited in subsequent generations of papers

• Download figure
• Open in new tab
• Download powerpoint

Policy implications

This study raises a number of important policy issues. Potentially the most important observation is in figure 2, which shows that authors of UK guidelines are citing UK papers. This may suggest that the guidelines are not citing all the internationally available evidence and that there is a publication bias. This explanation is unlikely, however, because the appraisal criteria specifically assess whether the identification, evaluation, and selection of evidence have been properly conducted and are unbiased. That said, it would be interesting to see if US guidelines cite US papers or whether in the United Kingdom a disproportionate volume of papers relevant to guidelines is published. Furthermore, guidelines informed by local (UK) research might not be a bad thing per se. Indeed, the preferential citing of UK papers may provide good evidence for supporting a local science base. If so, then the central policy question is: does a strong science base lead to better clinical practice? As far as we are aware, this assumption has never been fully tested. One way would be to link data on research evaluation (such as the number and quality of peer reviewed publications) with data on clinical audit (such as standardised mortality adjusted for case mix).

A second observation is the length of time it takes for basic research to flow into clinical practice. It has been shown that on average it takes three years for a paper to be cited,23 and figure 1 shows an average eight year time lag between the publication of a paper and the publication of a guideline. In other words, it would seem to take about 17 years (that is, 8+(3×3)) for the fourth generation research papers to feed into clinical practice. From a policy viewpoint, this raises the question of what the optimum time is for research to be fully evaluated and put into practice and whether this process needs to and can be speeded up.

A related issue is how basic research supports clinical research. Figure 4 shows that basic research is not being cited in clinical guidelines, and although the proportion increases by generation, only 8% of basic research is cited at the fourth generation of papers. It is hard to estimate how much basic research one would expect to be cited at the fourth generation, but it is significantly below the 25% for all UK biomedicine. Does this imply that there is little interaction between the clinical and basic research communities? Or is the information flowing through other routes, such as tacit knowledge, medical education, etc? Interestingly, these findings are at odds with those of past studies, which have concluded that basic research is an essential bedrock of clinical medicine.24 These studies and the current analysis, however, take a retrospective look—that is, they identify a success (such as a citation in a clinical guideline or a medical advance) and look at the supporting evidence. An alternative approach would be to identify a body of basic research published some time ago and follow its subsequent knowledge flow. This would further our understanding of how research feeds into medical (or non-medical) advances.

Methodological limitations

This study has shown how applied bibliometric techniques can be used to generate and inform several interesting policy issues in research and development and provide a way to evaluate the impact of research on clinical practice. There are, however, several limitations to our approach. Firstly, our analysis is based on “found papers.” These account for 70% of all references cited in clinical guidelines, and thus there could obviously be biases in our results if the papers we failed to find had significantly different characteristics to those analysed. Secondly, a citation in a clinical guideline does not guarantee an impact on health and therefore a payback on the research investment. Nevertheless, a citation in a clinical guideline could be considered as an indicator of research utility and thus an intermediate outcome, or “secondary output” in the language of the payback model. Thirdly, bibliometric analysis does not give any indication of the importance of a cited paper. This problem could be overcome by assigning weights to papers on the basis of quality, sample size, or such like. Finally, the analysis was undertaken in the absence of a denominator. Only the “successful” papers were examined, and we do not know how many papers have been published that are potentially relevant to the topics covered by the guidelines but were not cited in the clinical guideline.

Towards an evidence based research policy

Despite these limitations, we would strongly defend this type of analysis and encourage other investigators to spend some time thinking about the way research is managed. In one sense the lack of solid conclusions reflects the lack of evidence in this area. This is despite recognition in the BMJ in 1987 that “we need to research research”25 and the strong methodological synergy between epidemiology, especially in the analysis of observational datasets, and evaluation of quantitative research as presented in this paper. Indeed, in a period when researchers are demanding that clinicians practice evidence based medicine it is only appropriate and correct that such researchers audit and evaluate the research outputs and outcomes of their own investigations.