Assessment of research waste part 1: an exemplar from examining study design, surrogate and clinical endpoints in studies of calcium intake and vitamin D supplementation

Redundant research represents a large cost to society and misuses participants’ resources and altruism. Chalmers and Glasziou estimated that more than 85% of research might be wasteful because the wrong research questions are studied or because of poorly designed studies, inaccessible results, and biased reporting [1]. Very few studies have systematically examined research topics for evidence of waste, or established the methodology for doing so. In these companion reports [2], we assessed research waste in a single field - calcium and vitamin D research.

Calcium intake (through diet or supplements) and vitamin D supplementation have been very extensively researched as interventions for older adults for over 30 years. While undertaking recent systematic reviews [3‐8], we noticed that many studies in these fields seemed to be designed to replicate existing knowledge rather than to determine new clinically relevant findings. The large body of published research in these fields allows an assessment of one important aspect of research waste, the quantity of research conducted that was unnecessary because existing knowledge was extensive. In this first report, we use existing databases compiled from our recent meta-analyses, which collated all studies of increased calcium intake and all randomised controlled trials (RCTs) of vitamin D supplements. We assess whether each study with an endpoint of bone mineral density (BMD) or fracture constitutes research waste, i.e. an unnecessary duplication without addressing shortcomings in previous research [9]. We focus on study design (observational versus RCT) in calcium intake studies and type of endpoint (surrogate versus clinical) in vitamin D supplementation RCTs.

These results, obtained in a specific field of research, support the view of Chalmers and Glasziou that a very high proportion of published research is wasteful. For both calcium intake and vitamin D supplementation, both observational studies and RCTs with surrogate endpoints were far more prevalent than RCTs with clinical endpoints. Thus, observational studies of calcium intake outnumbered RCTs by at least 3 times, and for both calcium intake and vitamin D supplementation, studies using surrogate endpoints outnumbered studies using clinical endpoints by 1.6–3 times. For the skeletal endpoints of BMD and fracture, these imbalances persisted long after the tipping point when the hypothesis that calcium intake or vitamin D supplementation might alter fracture risk had been addressed and answered. Of RCTs of calcium intake or vitamin D supplementation and fracture or BMD published at least 5y after the tipping point, 36–46% were not novel, testing hypotheses about effects on BMD that had previously been established. Of those considered novel, 60–80% added no new clinical knowledge, and amounted to a third of the published RCTs. Almost all observational studies of calcium intake and BMD or fracture published 5y after the tipping point were not novel. Thus, we considered that since 2000, 73% of RCTs, 98% of observational studies, and 95% of clinical research on calcium intake and BMD or fracture was wasteful and that since 2005, 69% of RCTs of vitamin D supplements and BMD or fracture were research waste. In research on non-skeletal endpoints, the marked imbalances between observational studies and RCTs, and between RCTs with surrogate and clinical endpoints suggest that waste in researching calcium and vitamin D is not restricted to bone health.

In estimating that 85% of research represents waste [1], Chalmers and Glasziou focused on four aspects - the choice of research questions, quality of research, unpublished results, and the quality of research reports. Our findings are relevant to the first and second aspects. Chalmers and Glasziou estimated that 50% of studies are wasteful because they are unnecessary, poorly designed without reference to a systematic review, and/or have major methodological weaknesses [1]. Our findings suggest that this may be a considerable underestimate, at least in some research areas. Studies with weaker designs (observational studies or RCTs with surrogate endpoints) were far more prevalent than RCTs with clinical endpoints, but we considered almost all (90–98%) such studies published over a 15–20 year period to represent research waste. If our findings apply to other fields, the overwhelming majority of RCTs and observational studies published in mature research areas may be wasteful.

A strength of our study is the extensive systematic literature searches and categorisation of large, complete sets of observational studies and RCTs of calcium intake and skeletal health, and RCTs of vitamin D supplements published over a long period of time. We are not aware of previous attempts to categorise or quantify research waste, but our methods are very labour and time intensive, and require adequate resourcing if they were to be more widely utilised. Limitations include that only two research areas were studied and therefore the results might not apply to other fields, and that we estimated the scope but not cost of the waste. We also considered both study design and study endpoints, which we think makes any conclusions more compelling than if the research areas or the study design and endpoints were assessed separately. Assessing studies as novel, contributing to clinical knowledge, or representing research waste requires subjective decisions. In general, however, agreement between the independent assessments was high. Agreement was lowest for novelty because many studies were of highly selected cohorts not previously studied. Deciding whether small differences from previous cohorts (such as ethnicity, gender or age) are sufficiently unique to be considered novel is difficult. Replication of previous studies is essential, and while disagreements may occur as to whether a study represents replication or redundancy, the number of observational studies and RCTs with surrogate endpoints extended well beyond the number needed for replication. Research can take a long time from inception to completion, and it is possible that some studies were initiated before the tipping point but not published until more than 5y after the tipping point, and were then assessed as not contributing to current knowledge. The literature search includes papers published until September 2014. As a methodological study, it was an extremely comprehensive, but very time-consuming attempt to identify all relevant clinical studies (observational and RCT) for a large number of outcomes. This search took almost 12 months to complete, and so it was not practical to update it subsequently. We are not aware of an RCT of calcium or vitamin D with fracture as the primary endpoint reported since 2014. Based on the data from 2011 to 2014, we would expect more than 75 observational studies to have been published since 2014, but < 10 RCTs of calcium and > 200 RCTs of vitamin D supplements. These recent publications are therefore unlikely to alter our conclusions related to research waste in this field.

An important issue raised by this research is how to assess research waste. Widely accepted definitions of waste are not yet available, and an element of subjectivity will always be required. To minimise the impact of subjectivity, we followed approaches used for high quality evidence-based systematic reviews: we explicitly stated the criteria for studies to be categorised as research waste, independently assessed studies, documented levels of agreement, resolved disagreements by consensus discussion, and reported our judgements for each study. However, other researchers might classify studies differently, and differences of opinion are common in similar circumstances. For example, Cochrane researchers recently reported that 45/156 risk of bias judgements differed between two groups of authors [11]. Differences in definitions and study classification between researchers might provide insight into this issue and lead to better definitions of research waste, and better approaches to its assessment.

A related issue is when the tipping point can be considered to have occurred. In the design phase of any RCT, a literature review should be standard practice. Thus, trialists should be aware of the most recent evidence before the study protocol is finalised, and therefore whether RCTs with meaningful clinical endpoints should be undertaken, or whether more observational studies or RCTs with surrogate endpoints are still required. Researchers may have different views on the exact date the tipping point occurred, but the cumulative meta-analyses and publication rates of observational studies highlight that it was many years ago that the hypothesis of fracture prevention for both calcium and vitamin D supplementation was formulated. The exact date of the tipping point, and whether 5 years is sufficient to allow dissemination of that information may be open to debate, but, in our example, would not change the conclusion that the majority of recent published research has not addressed the issue of fundamental clinical importance- whether calcium or vitamin D supplementation prevents fractures.

Research waste and how it can be reduced is increasingly being considered an important topic [9, 12‐19]. The suggestions most relevant to our findings are the improvement of research priorities [9] and study design [15], in particular, performing systematic reviews to inform research proposals, and ensuring that study methods are optimum. Our finding that research waste continues unabated for long periods of time after a hypothesis has been generated or a research question has been answered suggests that most academics and funders have not so far corrected this problem. Publication of all RCTs is recommended, so if research has progressed to the stage of publication, it is difficult to argue that the primary RCT results should not be published. Considerable onus therefore necessarily falls upon assessments performed at the funding or ethical approval stage. These should be informed by a systematic review [9, 20], but that may not be straightforward. If systematic reviews exist, they often have discordant findings [21], and “living” systematic reviews updated with new evidence in real-time are not yet available [22]. Conducting a systematic review is time-consuming but necessary if one is not available or out-of-date. Finding relevant research is difficult because bibliographic database searches are non-specific, require screening of very many titles and abstracts, and frequently miss relevant studies. Involvement of information scientists and methodological expertise in the appraisal of funding applications might help to reduce research waste. This would require investment in appropriate resources but might improve the efficiency and accuracy of research funding.

We found that a very high proportion of observational research on calcium intake was wasteful. Many observational analyses are conducted opportunistically [23], and journal reviewers and editors could play important roles in declining manuscripts that examine established hypotheses that need, or have already been tested in, randomised studies, or where the findings are fragile and unlikely to be correct [24]. However, the large number of journals courting publications and academic incentives for publication quantity suggest that this strategy may be difficult to realise.

We identified a very high proportion of research waste in the fields of calcium intake and vitamin D supplementation for bone health after the need for RCTs with meaningful clinical endpoints had been established. This waste was largely due to continued publication of observational studies and RCTs with surrogate endpoints unable to address the relevant clinical questions. Nearly all the observational studies and nearly half the RCTs were not novel and another third of RCTs added no new clinical knowledge. Thus, collectively, 95% of recent RCT or observational studies of calcium intake and BMD or fracture, and 69% of recent RCTs of vitamin D supplements and BMD or fracture were research waste. Strategies to reduce research waste should be devised and studied to determine their effectiveness.