Main findings, implications and comparisons to existing research
While no associations were found between effect size and IF, reporting sources of funding, or conflicts of interest, there was strong evidence of a large association between absolute magnitude of effect size and the explicit reporting of ‘no funding’. We first discuss IF and then COIs and funding.
Impact factor
The results show no evidence that IF is associated with effect size reported in LBP trials. Effect size is much more variable in journals with low IFs and since journals with higher IFs tend to publish larger trials this likely explains the relationship between effect size variance and journal IF. Journal IF was not associated with direction of result, although there was some evidence that trials reporting a funder had a higher IF than those who did not report funding status.
Suñé
et al reviewed clinical trials evaluating drug therapy published between 1997 and 2004 and classified the outcomes of these trials as positive, negative, or descriptive (non-controlled) [
338]. They found no difference in IF based on trial direction, but they found the IF was significantly lower in trials classified as descriptive. Littner
et al found that over a five-year period in the field of neonatology, articles with negative results were more likely than articles with positive results to be published in journals with lower IFs [
339]. Penel and Adenis found the same pattern of association in phase II trials investigating anticancer therapies [
340]. Outside of the medical fields, Murtaugh has explored the relationship between standardised effects and IFs in published meta-analyses of terrestrial plant competition, predation in streams, woody plant growth under elevated
CO2, and marine nutrient enrichment experiments. Using raw data, he similarly applied weighted least squares regression analysis of study-specific means of the absolute values of the log response ratios on log of journal IFs and found some evidence that in two of the four areas studied (Nutrient enrichment experiments and predation in streams) that journal IF was associated with reported effects [
341].
The presence or absence of associations differs across different research areas. It may be that there is less competition in high-impact journals in terms of newsworthiness of LBP trial results relative to other fields. As it is rare for individual treatments for LBP to stand out dramatically from others in terms of effect size, effect sizes in LBP trials may not be a big driver of an acceptance decision in higher-impact journals.
COI and funding status
We found no evidence that COI category or reported funding status is associated with effect size reported in LBP trials. However, we observed that absolute magnitudes of effect sizes tended to be about one SD larger for trials that declared no funding compared with trials that did not report funding status. The observed association is not due to confounding by sample size. Jacob Cohen, who originally defined standardised effect sizes, considered effect sizes of 0.2 or less to be small, 0.5 to be medium, and 0.8 and above to be large [
20]. Using Cohen’s categorisation, the effect size in the larger trials of interventions for LBP tend to be only small-to-medium in magnitude [
7].
This relationship is in marked contrast to that observed in other fields, where evidence suggests industry-funded, industry-linked studies, or studies with an industry-funded author, report greater effect sizes than independently funded studies [
11,
12,
342,
343]. In the authors’ experiences, LBP research trials tend to be more commonly funded by government and charitable organisations rather than by industry. It may be that, in the case of LBP trials, reporting larger effect sizes, may be higher amongst studies with fewer resources.
Our a priori approach was to compare categories of explicitly reporting no funding/COIs, and explicitly reporting funding/COIs with not reporting anything about funding/COIs, and it is these results that are reported. As a post hoc comparison to explore reporting of funding further, we compared trials that explicitly reported having funding with trials that explicitly reported not having funding, and found strong evidence of a large effect (β=−0.89 (95 % CI −1.46 to −0.33), P = 0.002), suggesting that those reporting receiving funding, report considerably smaller effect sizes than those reporting their trials were not funded.
Trial quality may partially explain the results. It has been previously shown that larger trials in non-specific LBP tend to be higher quality [
17,
344]. We did not explore trial quality in our study. Another consideration may be that pragmatic trials tend to be done more often in LBP research, since many interventions under assessment are complex in nature and pharmacological interventions (which are usually of efficacy rather than effectiveness) [
345] are comparatively rare. It may be that trials more toward the pragmatic end of the spectrum, which may be more difficult to do in the absence of funding given their typical requirement to be large in scale, may be associated with smaller effect sizes simply because the comparator is often another active intervention. Conversely, efficacy trials may have higher effect sizes in part due to more commonly utilising placebo/sham comparisons. We did not explicitly set out to explore this. However, as another
post hoc comparison we looked at the intervention comparisons in our included trials, and those that were compared to sham/placebo had an effect size of 0.74 (large), in contrast with those compared to a non-sham/placebo interventions, which had an effect size of 0.29 (P = 0.077;
i.e weak evidence of a small-to-moderate difference).
Strengths and limitations
Meta-regression modelling is most useful in this case as a tool to assess the role of chance in the observed results. We caution against use for prediction, since epistemologically this may not be entirely sensible: prediction may involve a reversal of the direction of causality; authors likely choose journals on the basis of publishing work they believe to be newsworthy, high-quality, or of interest to a particular journal’s readership. More robust and simpler solutions to establishing the role of chance, and whether relationships between effect size and IF are monotonic could be used (such as non-parametric correlation) but, as Murtaugh points out, such approaches are less able to incorporate study-specific weights and are ultimately less powerful [
341]. Also, such approaches are not as conducive to the inclusion of covariates. In this study we had sufficient power to detect a medium-to-large effect size in terms of funding category, but not in terms of COI, which were only reported in 7 % of trials.
COIs disclosure may or may not be insisted upon by a journal, or COI forms may have been completed but not reported with the article. Additionally, disclosed COIs may or may not be relevant to the trial. We explored only the presence or absence of such statements reported with the article and did not judge the relevance of disclosed COIs to the trial, nor whether the publishing journal required disclosure, and this as a limitation of our study.
The large I2 values for the models suggests that the residual variance explained by heterogeneity is very high. This is to be expected since the included trials featured many different interventions. In our analysis, other than having a detrimental effect on power, the high I2 is inconsequential to interpretation and does not present a limitation as it would in a meta-analysis of a specific treatment effect. We were not focused on estimating the effect of a specific intervention, but the association between effect size and IF, COIs, and funding across many different interventions for nsLBP, some of which will naturally have larger effects than others.
We imputed zero values for IF in the case of journals without an official IF. Many journals use unofficial IFs and including these could have been used to introduce more information into our models. We reasoned that the majority of journals without official IFs would likely have unofficial IFs of less than 1.00 and preferred to use only official values. We note that if IFs had been associated with effect size then our estimates may have been exaggerated. As we did not find any association with effect size, imputing values where there was no official IF was of limited consequence and does not affect conclusions.
In attempting to explain our results, we have hypothesised that there may be a link to study quality, which we did not explore. While there is some evidence of a small effect of poor quality on effect size in LBP trials from other work, we would welcome future investigations using the Cochrane Risk of Bias tool, since the judgement criteria in this tool can be applied to either pragmatic or efficacy trials without prejudice. Lower quality trials may have been associated with both absence of definition of primary outcome measure, where we would have used outcome measure selection method 3 or 4 (see Methods section), as well as with larger study effects. While we recorded and reported authors explicitly identifying an outcome as primary, for the trials in which this was not explicitly identified, we did not record how often primary outcome identification method 2, 3, or 4 needed to be used. So we caution that there may have been an unmeasured confounding factor.
We rejected trials from which we could not abstract population-specific SD data required for the meta-analysis. This resulted in 47 rejections and opens a possibility for bias, in the case that not reporting these data is also associated with reported effect size. While not specifically an item in the CONSORT statement, this is something one might reasonably expect to be discussed within a sample size calculation. For this reason, we suggest the absence of its reporting, is more likely to be associated with lower quality. Assuming this, and the premise that lower quality trials report larger effect sizes notwithstanding direction, are both correct, then our results will tend toward being conservative.
Finally, we restricted our systematic review to three large databases, reasoning that these index the majority of nsLBP RCTs. We acknowledge however, that the review of the period is unlikely to be exhaustive and that there may be further associations between trials indexed in other databases alone, and quality; and thus with the potential to alter results. However, our results cover most of the field and therefore provide a useful account of behaviour.
Recommendations
Based on our results we recommend that journal editors consider giving increased scrutiny at peer-review stage to unfunded LBP trials. Researchers need to carefully consider whether the trial in question can be adequately and appropriately conducted in the absence of funding, and whether the protocol should be subject to peer review. Consumers of LBP trial reports should note this relationship in the case trials are unfunded.
The causal pathway for the relationship between funding and effect size needs further exploration. If larger effect sizes yielded by unfunded trials are incorrect, these may add noise to data consumed by review work decreasing the precision of meta-analyses. If internal validity is a factor, one might raise the question of whether it is ethically justified to undertake unfunded trials of interventions for LBP. If the extent of the pragmatism of a trial is a driving factor, then absorption of the higher absolute effect sizes into specific review work is less of a concern, but a scale of pragmatism might aid interpretation of effect size from individual trials and be useful to reviewers.
Research into the relationship between funding status and effect size, and IF and effect size, appears to be dependent of the field of research and the nature of interventions under investigation. For this reason we suggest that investigations are conducted across different fields and interventions so that the relationships between COIs and funding and effect size can be better understood and consumers can take this into consideration, as appropriate.
Authors of LBP trials should explicitly report whether or not funding was attained, as only around two-thirds of all authors are currently doing this. Moreover, more authors need to be explicit about whether or not there were COIs as in our dataset only 29 % of authors are doing this. Journals and editors could consider taking steps to ensuring this information is reported.