Introduction
In response to the COVID-19 pandemic, researchers initiated thousands of randomised trials seeking possible treatments. Many of these trials have been registered, conducted and published but the amount of research and the speed at which it was produced created an infodemic making it difficult for potential users of the research to discern what information is relevant, accurate and current [
1]. Furthermore, the rush to conduct trials meant that many had methodological shortcomings, such as small sample sizes and lack of blinding or allocation concealment [
2]. This, coupled with unnecessary duplication, has led to a high “noise to signal” ratio in the COVID-19 evidence base and research waste [
3].
Research waste can occur during any stage of the research process [
4]. In the production phase of research, it occurs when the question is irrelevant to clinicians or patients or has already been answered definitively. Chalmers et al. [
5] highlighted that new research should not be started unless existing research cannot answer the question adequately and systematic reviews are one way to determine this [
6]. For example, through 2020 and 2021, several trials of chloroquine or hydroxychloroquine (HCQ) for treating COVID-19 were published, and a Cochrane systematic review in February 2021 concluded that HCQ has little to no effect on the risk of death and that further trials of HCQ or chloroquine should not be carried out [
7‐
10]. Further, research waste can arise because a lack of coordination and collaboration can lead to unnecessary duplication of research which will add limited evidence strength. For example, even though larger, multisite studies recruiting thousands of patients might be underway, multiple small single-site studies still often take place. This can create waste [
2]. In the reporting phase of research, waste is generated when published studies have unusable or biased results. Under-reported or unpublished research also contribute to research waste and breach the researcher’s ethical obligation to make results of research on humans publicly available [
5,
11].
Research waste stems from research with little to no societal, educational or stakeholder benefit [
4]. Poorly reported or unpublished results compound research waste, pose a risk to the care of future patients and present ethical concerns [
5,
12]. Although research waste is not a new problem, it was accelerated by the COVID-19 pandemic and was prevalent across trials, evidence synthesis and guidelines [
2,
13,
14].
Publication of results helps limit research waste by ensuring that studies add to the knowledge pool regardless of their results [
15]. Traditionally, publication was achieved by publishing in a peer-reviewed journal but many trial registries, which help to improve transparency about trial methods, also now provide a facility for researchers to upload their results [
16]. Trial registries that meet the requirements of the World Health Organization (WHO) and International Committee of Medical Journal Editors (ICMJE), are searchable through the International Clinical Trial Registry Platform (ICTRP), allowing access to information on hundreds of thousands of randomised trials [
17]. It has long been known that the results of trials influence whether or not they are published but the information available in trial registries can also be used to investigate whether any design characteristics of a trial are associated with the publication of its results [
18].
The large number of registered trials of treatments for COVID-19, the global nature of the research and the speed at which research took place, provide an opportunity to assess trial characteristics that may lead to successful publication in peer-reviewed journals. Therefore, we conducted this meta-epidemiological cohort study to determine the proportion of registered randomised trials of COVID-19 treatment that were published in peer-reviewed journals, or on preprint servers or shared their results publicly in other ways; to describe the characteristics of these trials across three possible treatments (corticosteroids, hydroxychloroquine and vitamin D); and to examine the association between trial characteristics, publication status and research waste.
Methods
We conducted a meta-epidemiological cohort study of randomised trials of hydroxychloroquine, corticosteroids and vitamin D, identified on ICTRP that were registered up to 31 December 2021. These three possible treatments were chosen to limit the scope while still exploring a variety of research, allowing us to include a treatment for which there is now a high certainty of evidence of no mortality benefit, hydroxychloroquine (both hydroxychloroquine and chloroquine were considered as one treatment form); a treatment for which there is high certainty of evidence of a mortality benefit, corticosteroids; and a treatment for which the effects are still uncertain and controversial, Vitamin D (both Ergocalciferol and Cholecalciferol were considered as one treatment form) [
10,
19,
20]. We also assessed how many trials had been registered after key systematic reviews had been published which provided high certainty evidence of the effects of the treatment on mortality. We used an
a priori protocol (which was established before the data search was done and submitted to the Stellenbosch University Health Research Ethics Committee and received ethics exemption: X22/09/003_COVID-19) and used a flow diagram to present our findings which is based on the PRISMA guidance [
21].
Search methods and screening
We searched ICTRP for randomised trials of treatments for COVID-19 which had been registered between 1 November 2019 and 31 December 2021. A separate search was conducted on ICTRP for each of the three treatment groups using a combination of keywords and free text words (Additional file 1). The searches were restricted to COVID-19 using ICTRP’s preloaded filter and all trial phases were included. No other filters or eligibility criteria were applied in the search. The search was completed on 13 September 2022, allowing eight months from the end of the registration date eligibility criterion for the transfer of information to ICTRP. Results were downloaded into a Microsoft Excel spreadsheet for screening. During screening, eligibility criteria were applied by one author (LF) for final inclusion.
Eligibility criteria
We included registered trials if they met the following criteria: [
1] registration (prospective or retrospective) on the trial registry before 31 December 2021; [
2] the primary purpose of the trial was treatment of COVID-19; [
3] study design was a randomised trial; and [
4] the target intervention in the trial was (or included) one or more of the following: hydroxychloroquine, corticosteroids or vitamin D. We did not limit by country, language or treatment setting. We excluded registered trials if [
1] the primary purpose was supportive care or prevention; [
2] it was quasi-randomised or not randomised; and [
3] it was not performed in humans.
All available trial characteristics were extracted from the ICTRP data (Additional file 2). Our primary outcome of interest was the proportion of registered trials that were published, or had reported their results publicly in another way, but publication status was not an available field in the ICTRP data. Consequently, a public platform search was conducted to follow up each trial in this cohort. The search was done on PubMed and Google Scholar between 10 and 20 October 2022 using primarily the trial identifier. When this search did not yield a record, the registered title was used in the search, and retrieved records were checked against the authors and the country of origin. If we found the results of the trial in the public domain, we regarded it as “published”. If it was not found, it was classified as “not published” and did not contact those responsible for registering the trial for further information. We categorised publication status as follows: [
1] published in a peer-reviewed journal, [
2] published on a pre-print server, [
3] mention of available results on ICTRP, and [
4] other (such as a research letter or retraction). When the search found more than one such record for a trial, we used the most up-to-date record. For example, if a trial was published in a peer-reviewed journal and then later retracted, we counted the retracted record. Current trial status was not available in the ICTRP data, so we searched for this information in the relevant primary registry for each included trial and categorised it as unknown, ongoing, complete and terminated/suspended. The ongoing category included trials that were yet to start recruiting, busy recruiting, active or ongoing according to the primary trial registry. We translated the necessary information from the primary trial registries if it was not in English.
Data management and coding of trial characteristics
The characteristics’ terminology was largely based on the ICTRP terminology, including phase of trials. Although ICTRP, and many of the primary trial registries, have a primary sponsor field, they do not explicitly state who funded the trial or the source of any financial support. The primary sponsor is defined as the individual or organisation that takes responsibility for the initiation and management of the trial, which may include financial responsibility [
22]. Based on their main purpose or the description on their website, we divided the primary sponsors into the following groups: [
1] academic, [
2] research institute, [
3] government, [
4] medical centre, [
5] pharmaceutical company, [
6] organisation, [
7] other and no sponsor.
Data analysis
We conducted a descriptive analysis of the trial characteristics in STATA 16. We explored the association of individual trial characteristics with publication status using a univariate regression analysis, presenting odds ratios (OR) and confidence intervals (CI). When each level of publication status was compared across the three treatments, we found that the results were homogenous with overlapping confidence intervals and therefore pooled the data from all three treatments to maximise power in the regression models. We further investigated the relationship between trial characteristics and publication status using a multivariable logistic regression model. Trial characteristics associated with publication status in the univariate test (p ≤ 0.2) were included in the logistic regression model, for which P-values ≤ 0.05 were deemed statistically significant for the final model.
Discussion
Among the characteristics we examined, we found that an increased odds of publication was associated with medium or large target sample size; and that having a medical centre, the government or a research institute as the primary sponsor was associated with lower odds of publication. Similarly, in other areas of health research, intervention type and particular types of study sponsors, specifically a pharmaceutical company, have been found to significantly influence publication status [
23]. Although it has been shown that some trial characteristics influence the quality and risk of bias for a trial, such as the use of blinding and allocation concealment, these characteristics were not found to be associated with trial publication status in our study [
24]. The assumption that a multinational trial or one with more than two randomised groups would have more key role-players and therefore more support and a higher chance of being published was also not demonstrated by this study.
In other health fields, such as oncology, the publication of non-significant results is a challenge, even for bigger trials [
25]. At the height of the pandemic, journals were inundated with submissions about COVID-19 and were having to process manuscripts that might be reporting research that would make little or no contribution to the evidence base because of the rapidity with which some areas were developing, which might have been particularly problematic for the large number of small trials as the results of larger, more definitive trials became available.
As far as we know, this study is the first to compare publication status of registered randomised trials of treatments for COVID-19 with trial characteristics revealing a concerningly low publication rate of COVID-19 trials. In other areas of health, similar trends were seen, for example one study reported that less than half of trials registered on
clinicaltrials.gov had been published, while another reported that only 28.6% of the registered trials were published within 24 months of trial completion [
26,
27]. Our results shed some light on this, as larger trials were more likely to be published, and potentially appropriately funded to publish and afford publication costs, compared to smaller singe centre trials. More precise trial sponsor and funding data is required in trial registries as these are potentially important predictors of research waste and necessary information for researchers, funders and policy makers.
On ICTRP, only 20 studies reported that their results were available, and of these seven had published in a peer-reviewed journal. However, our search found a total of 77 (22%) of the trials in our sample had been published in a peer-reviewed journal. Trials classified as “completed” accounted for 72 (20%) of the eligible trials in this study and we found that 25 of these had been published in peer-reviewed journals and seven had shared their results publicly in some other way, leaving 40 “completed” trials with unshared results. Furthermore, we found that 39 of the 77 studies that were still classified as “ongoing” on the primary registry had been published in a peer-reviewed journal. In some cases, this was due to the publication of interim results of a trial or partial results for one treatment group in the case of platform trials, but this seems unlikely for all these 39 trials, and we expect that many have completed without updating their registry entry. This reinforces that there is a problem with a lack of updating of the trial registry by researchers.
The “ongoing” category made up 44% of the trials in our study, and included recruiting trials, trials that began towards the end of our timeframe for eligible registrations (31 December 2021), trials that began earlier but were taking longer and trials that may have been completed but without an update to the registry entry. Some of these trial entries have not been updated for over 12 months. It is possible that an important proportion of these are in fact no longer ongoing. This highlights the importance of researchers updating the trial registries on a regular basis to ensure the status of their trial is transparent and their results known, especially if the trial is no longer ongoing.
A limitation of our study is with the search for publications. For instance, if the title used for a trial in its registry entry was not used in its publication or the published article did not include the trial identifier, it would not have been found during our search. Furthermore, a search beyond Google Scholar and PubMed might have found more publications. It would also have helped those searching for trial results if the published reports of all trials included the trial’s registration ID, which would help to reduce waste by making it easier for users to find the research. In addition to the extent of our search, a further constraint is that only one author performed the search, screening and data extraction.
Another potential limitation of our study is that we collected most of the data from ICTRP. As ICTRP draws its data from multiple registries it creates an entry for each unique trial ID. If a research team register their trial on more than one registry and do not list the same title and research team, this may lead to duplicates which we were unable to rule out. Each entry was therefore treated as a separate trial. Moreover, some loss of data detail could have occurred because of the information transfer from the primary registry to ICTRP. However, when we reviewed primary registries for key variables, we found gaps in some of these. Trial registries vary in the registration information and updating requirements which has an impact of the quality and availability of this data and possibly on publication. We recommend that trial registries have firmer guidelines and updating policies and that trialists are specific as possible when uploading and updating information in the registry.
Conclusions
We performed a meta-epidemiological cohort study to describe the characteristics and publication status of registered randomised trials of the use of corticosteroids, hydroxychloroquine, and vitamin D to treat patients with COVID-19 and to analyse the association between trial characteristics and publication status. Our findings suggest that target sample size of above 100 patients is an important predictor of publication. This may be related to other factors such as funding or the institutions involved in a trial but highlights the need for researchers to collaborate on large, definitive trials to increase their chances of publication and making an important contribution to the evidence base. We found that a large proportion of registered trials have not yet made their results publicly available. Even for trials not published, if the results are not made publicly available at minimum, this amounts to research waste. Our findings also identified a discrepancy between the trial’s status of some trials in the registry and their publication status, with publications for some of those marked as “ongoing” on the registry having been published and likely to be completed.
In order to reduce research waste and to ensure that the rush to do research during the COVID-19 pandemic does not lead to the “equivalent of the boxes of useless aid supplies that end up rotting away on runways and in warehouses after large scale disasters and humanitarian emergencies” [
3] we recommend that trial registries are updated more frequently and that results of trials are made public. We also recommend that further investigations are done to assess for factors contributing to publication status and research waste and that these factors are carefully considered before studies are designed, funded, granted ethical or regulatory approval and initiated both in further emergencies such as the COVID-19 pandemic but also in health care in more normal times.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.