Abstract
Context
The misrepresentation and distortion of research findings, known as “spin,” has been shown to affect clinical decision making. Spin has been found in randomized controlled trials (RCTs) published in various fields of medicine.
Objective
To evaluate the abstracts of RCTs found in the cardiology literature for spin.
Methods
The authors searched PubMed using a specific string of keywords to identify previously published articles documenting RCTs of cardiovascular treatments in humans. To be included, a cardiology trial had to randomize humans to an intervention, statistically compare 2 or more groups, and have a nonsignificant primary endpoint. Records were excluded if they did not meet these criteria. Data extraction was double-blinded and done using a pilot-tested Google Form. Items extracted from each trial included the title, journal, funding source, comparator arm, primary endpoint, statistical analysis of the primary endpoint, secondary endpoints, statistical analysis of secondary endpoints, and trial registration number (if reported). The 2 authors who screened records for inclusion were then asked whether spin was present in the abstract of the randomized trial. Spin in the title, abstract results, abstract conclusions, and selection of reported endpoints were considered.
Results
Of the 651 PubMed citations retrieved by our search string, 194 RCTs with a clearly defined primary endpoint were identified. Of these 194 RCTs, 66 trials contained nonsignificant primary endpoints and were evaluated for spin. Of these trials, spin was identified in 18 of the 66 abstracts (27.3%).
Conclusions
Spin was present in our sample of cardiology RCTs. Spin may influence clinical decision making by creating false impressions of the true validity of a drug or intervention.
In cardiology, the accurate reporting of results from randomized, controlled trials (RCTs) are essential to evidence-based clinical decision making. Biased reporting in RCTs has been shown to skew how physicians interpret findings; for example, the effect of biased reporting on physician interpretations of drug efficacy was examined in the SPIIN trial,1 in which 300 oncologists were asked to read an abstract with nonsignificant results. The intervention group received a version of the abstract that contained “spin” (biased reporting in the form of misrepresented or distorted findings, or in this case, misleading conclusions), whereas the control group received an abstract without spin. Before reading the full text of the article, the interventional group concluded that the drug was more effective despite the nonsignificant results. Authors may use spin to draw misleading conclusions about their study as an attempt to persuade readers that their results were significant when, in fact, they were not.
Clinical trial abstracts play an important role in the medical literature. They provide a synopsis of the full trial report and are meant to guide the reader quickly through the background, methods, findings, and conclusions of the study. However, because of the brevity of most abstracts, authors may be forced to choose which results to include and which conclusions they wish to convey. Consequently, abstracts commonly have evidence of spin.2 A previous study showed that internists frequently only read the abstract of a clinical trial, either to quickly browse the literature or screen out uninteresting reports.3 If cardiologists emphasize abstracts in the same way, they may be susceptible to spin. Further, if cardiologists, also like internists, are more likely to read the full text of a trial with optimistic conclusions, they may be exposing themselves to additional spin, since the full text of RCTs has shown similar rates of misrepresented research findings.2
For example, a 2011 evaluation of primary prevention trials of implantable-cardioverter defibrillators (ICD) found evidence of spin.4 In these trials, the background information provided was often imbalanced, focusing more on the benefits and minimizing the harm. In interpreting the data from trials, the authors again emphasized the benefits and failed to adequately discuss the adverse events associated with the devices, resulting in misleading conclusions. Even if ICDs were as effective as the authors concluded, failing to balance the background and discussion is a rhetorical strategy of spin that likely affects how cardiologists understand the benefits and risks of ICDs.
The primary objective of this investigation was to assess the frequency and manifestations of spin in the abstracts of cardiology RCTs. The secondary objective was to determine whether the presence of a funding source is correlated with the presence of spin in RCT abstracts.
Methods
This study was conducted in accordance with a previously written protocol that is publicly available via the Open Science Framework.5 We searched PubMed/MEDLINE on May 15, 2018, for reports of RCTs published in 2017 for cardiovascular treatment in humans. Our search protocol was as follows:
(((((((("J Am Coll Cardiol"[Journal] OR "Circulation"[Journal]) OR "Eur Heart J"[Journal]) OR "Am J Cardiol"[Journal]) OR "Int J Cardiol"[Journal]) OR "Heart"[Journal]) OR "N Engl J Med"[Journal]) OR "Lancet"[Journal]) OR "JAMA"[Journal]) OR "Ann Intern Med"[Journal] AND (Clinical Trial[ptyp] AND ("2017/01/01"[PDAT] : "2017/12/31"[PDAT])).
We then used the h5-index ranking in Google Scholar Metrics to rank journals in our sample. The h5-index is the h-index for articles published in the last 5 complete years—in this case, 2012 to 2016—and it represents the largest number h such that h articles published in 2012 to 2016 have at least h citations.6 The h5-index range for our included journals was 73-365. We then selected the 6 cardiology journals with the highest h5-index. Additionally, we selected the top 4 general medicine journals based on a combination of h5-index rankings and their history of publishing cardiology clinical trials in the past, which was assessed subjectively by our the perceived likelihood that these journals contained cardiology articles based on article titles. We searched using Print International Standard Serial Number because it eliminated the possibility of including journals based on keywords such as The Lancet: Oncology when searching for The Lancet. Search results were added to a PubMed collection and exported to Rayyan7 for screening by title and abstract.
Two authors (C.C. and M.K.) screened records for inclusion and extracted data for spin. Prior to the start of screening for inclusion and data extraction, both authors underwent training by another author (C.W.) to recognize and analyze spin. We drew heavily from the Boutron et al2 seminal publication on spin in RCTs with nonsignificant endpoints to develop training material, define spin, and devise extraction forms.2 We used the Boutron et al2 definition of spin in randomized trials with nonsignificant primary endpoints, which described spin as the “use of specific reporting strategies, from whatever motive, to highlight that the experimental treatment is beneficial, despite a statistically nonsignificant difference for the primary outcome, or to distract the reader from statistically nonsignificant results.” Training videos based on the Boutron et al study2 were created by 1 author (C.W.) to define spin and explain the screening and data extraction procedures. Each screener (C.C.) and extractor (M.K.) watched these videos and then underwent in-person training by another author (C.W.) to ensure understanding and calibration between partners.
To be included, each cardiology trial had to randomize humans to an intervention, statistically compare 2 or more groups, and have a nonsignificant primary endpoint. Records were excluded if they did not meet these criteria. Figure 1 shows a PRISMA flow diagram of this study's included and excluded trials. Data extraction was done using a Google Form that was pilot-tested 3 months before the study at Oklahoma State University. The pilot-testing of this Google Form was performed by 2 authors (C.W. and M.V.) to increase the feasibility of this study. Two other authors (C.C. and M.K.) independently completed this Google Form by extracting items for each included trial and were blinded to each other's answers during this period. Items extracted from each trial were the title, journal name, funding source, comparator arm, primary endpoint, the statistical analysis of the primary endpoint, secondary endpoints, the statistical analysis of secondary endpoints, and trial registration number (if reported). The same 2 authors (C.C. and M.K.) also responded in the Google Form whether spin was present in the abstract of the randomized trial. Spin in the title, abstract results, abstract conclusions, and selection of reported endpoints were considered. We considered evidence of spin to be present if trial authors focused on statistically significant results, interpreted statistically nonsignificant results as equivalent or noninferior, used favorable rhetoric in the interpretation of nonsignificant results (eg, “trend toward significance”), or claimed that an intervention had a benefit despite statistically nonsignificant results. If authors focused on statistically significant results, we catalogued whether they used within-group comparison, subgroup analysis, statistically significant secondary endpoints, or modified treatment population. All other strategies of spin that were apparent, but did not fall under one of the above categories, were recorded as we extracted data.
PRISMA flow diagram detailing the search and selection process in this study. Abbreviation: RCT, randomized, controlled trial.
The results from each author's blinded extraction were then exported from the Google Form to a Microsoft Excel sheet. Two authors (C.C. and M.K.) then met to discuss discrepancies between their answers. If there was a disagreement on any single item within a trial, the authors researched the associated trial and resolved the discrepancy through discussion. A third author (C.W.) was available as a third party adjudicator if agreement could not be reached between the original 2 authors. Summary statistics (frequencies and proportions) were calculated using Google Sheets. Unadjusted odds ratios were calculated using STATA 13.1 (StataCorp, LLC).
Results
General Characteristics
Of the 652 PubMed citations retrieved with our search string, 194 RCTs with a clearly defined primary endpoint were identified (Figure 1). Of these 194 RCTs, 66 trials contained nonsignificant primary endpoints and were evaluated for spin. A majority were RCTs testing a pharmacological intervention against an active treatment (40; 60.6%). Some RCTs (24; 36.4%) compared an intervention against a placebo or sham procedure. A funding source was not listed in 17 of 66 RCTs (25.8%). Of the studies that listed a funding source (49), industry funding was the most common and occurred in 19 (38.8%). Trial registration numbers were included in the majority of included RCTs (55; 83.3%). The primary endpoint was clearly stated in the abstracts of 46 included articles (69.7%).
Primary Endpoint
Spin was identified in 18 abstracts (27.3%). Spin was present in the title of 1 abstract (1.5%); this title insinuated clinical benefit for the drug intervention, despite no benefit being demonstrated. Spin was found in the results section of 7 of the included abstracts (10.6%), most often (3 of 7 abstracts; 42.9%) related to emphasis on a statistically significant subgroup analysis. Focusing on a significant secondary endpoint was another form of spin found in the abstracts’ results sections (2 of 7 abstracts; 28.6%). A complete summary of spin strategies used in the abstract results sections can be found in Table 1.
Sources of “Spin” in Reports of Randomized, Controlled Trials in Cardiology (n=66)
| Spin | No. (%) |
|---|---|
| Spin in the title | 1 (1.5) |
| Claim/insinuation of benefit when none exists | 1 (100) |
| Spin in the results | 7 (10.6) |
| Focus on (+) subgroup analysis | 3 (42.9) |
| Focus on (+) secondary endpoint | 2 (28.6) |
| Selective focus on (+) within-group comparison for primary endpoint | 1 (14.2) |
| "Trend toward significance" // "numerically longer survival" or similar verbiage | 1 (14.2) |
| Spin in conclusions | 16 (24.2) |
| Claim of benefit based on (+) secondary endpoint | 4 (25) |
| Claim of equivalence/non-inferiority versus control for a (-) endpoint | 4 (25) |
| Other | 3 (18.8) |
| Claim of benefit based on (+) subgroup analysis | 2 (12.5) |
| Focus on another objective | 2 (12.5) |
| "Trend toward significance" // "numerically longer survival" or similar verbiage | 1 (6.3) |
(+) indicates a positive focus by the authors (ie, focusing on a positive subgroup/secondary outcome)
(−) indicates a nonsignificant finding for an endpoint within the trial
Spin was found in the conclusions section of 16 of 66 abstracts (24.2%). Of these 16, evidence of spin was most often the result of claiming a benefit based on a significant secondary endpoint (4; 25.0%). Claiming equivalence/noninferiority versus control for a nonsignificant endpoint (4; 25.0%) was equally common. A complete summary of spin strategies used in the conclusions section of the abstracts analyzed in this study can be found in Table 1.
Secondary Endpoints
Spin and Funding Source
Industry-funded trials (19; 28.8%) were more likely to have evidence of spin in the abstract (unadjusted odds ratio, 6.5; 95% confidence interval, 1.2-43.6). Seventeen of the 66 studies (25.7%) did not mention a funding source, and spin was detected in 7 of those 17 (41.2%). Of the studies that mentioned a funding source, industry-funding was the most common (19; 38.8%). Of the 19 studies that were funded by industry, 8 (42.1%) had evidence of spin. A complete summary of abstracts with spin and their corresponding funding sources can be found in Table 2.
Sources of Funding in Reports of Randomized, Controlled Trials in Cardiology (n=66)
| Source of funding | No. of studies (% of total) | No. of studies with spin (% of total) |
|---|---|---|
| Industry | 19 (28.8) | 8 (12.1) |
| Not mentioned | 17 (25.8) | 7 (10.6) |
| Public | 16 (24.2) | 0 (0) |
| Other | 7 (10.6) | 3 (4.5) |
| Hospital | 7 (10.6) | 0 (0) |
Spin and Trial Registration
Of the 66 included RCTs, 55 (83.3%) reported a trial registration number. All endpoints from the included RCTs matched the registered endpoints. There was no evidence of spin due to selective reporting bias.
Discussion
This investigation found that 27.3% of cardiology RCTs with a nonsignificant primary endpoint had evidence of spin in the abstract. Furthermore, industry-funded studies were statistically more likely to have evidence of spin. Spin was most frequent in the conclusion sections of abstracts, and authors most often emphasized statistically-significant secondary endpoints or wrongly interpreted a nonsignificant P value as supporting equivalence between the intervention and control. The implications of these findings are clear: misrepresented research findings may mislead cardiologists. These findings are concordant with the results of previous studies.2,8-10
In the first evaluation of spin in the medical literature, Boutron et al2 found evidence of spin in 49 out of 72 abstracts, with the conclusion section being the most common area of appearance (24; 33.3%). Chiu et al8 performed a systematic review of spin comprising 35 published studies. According to their review, spin occurred in a median of 57% of abstracts. When they examined only trials with nonsignificant primary endpoints, as in our investigation, the median rate of spin in abstracts was 60.5%. Chiu's extensive systematic review included assessments of spin across multiple areas of medicine. In addition, Jellison et al9 reviewed the abstracts of 116 psychiatry and psychology articles. They found spin to be present in 56% of these articles, and concluded that all journal editors should be aware and continue to monitor for spin to reduce the risk of reporting bias. In 2018, Cooper et al10 conducted a cross-sectional investigation analyzing spin in the abstracts of 47 articles among the top-ranked otolaryngology journals based upon index-factor ranking through Google Metric Scholar. These authors found spin to be present in 70% of the investigated articles and concluded that spin may create false impressions about the true validity of a drug or intervention.
The potential consequences of spin are far-reaching. The busy physician may have difficulty detecting spin and may be more likely to recommend an unproven treatment if spin exists.1 Further, residents and physicians may make clinical decisions based on RCT abstracts alone, either because they lack time or lack access to full text of RCTs.11,12 Even when physicians have time and access to the full text of articles, they may only read the abstract to screen out uninteresting results or digest research findings.3
The implications of spin do not end there. Large scientific conferences are typically covered extensively by the media, and the dissemination of distorted research findings can have consequences beyond clinical decision making. Yavchitz et al13 found that spin in the abstract of an RCT increased the risk of spin in press releases and other media reporting. Phillips et al14 found that journal articles covered by the New York Times had 73% more citations than control articles in the first year after publication and that disparity in the number of citations continued in each of the 10 years after publication. The findings of these 2 studies show that spin in RCT abstracts affects public perception of medical interventions.
To illustrate the effect of media on public perceptions of medical interventions, consider an investigation of 675,000 Danish adults who began statin therapy between 1995 to 2010.15 Nearly 2000 statin-related news stories were reported during that time period, and 110 of them were deemed negative. Study investigators found that patients were 9% more likely to discontinue statins when negative news stories came out and 8% less likely to discontinue them when there was positive news coverage. After following patients through 2011, the investigators found that the patients who discontinued statins had a 26% increase in heart attack incidence and an 18% greater risk of death than those who remained on statin therapy.15
Solving the problem of spin requires exposure, awareness, and education. If cardiologists are unaware of spin in RCT abstracts, they will be unaware of the influence spin can have on clinical decision making. Even if awareness of spin is widespread, it may still affect cardiologists’ perceptions of drug efficacy and safety. This effect may especially be present if peer-reviewed RCTs with spin are not critically reviewed by the public. Passively reading the abstract of RCTs3,12 may result in accepting distorted research findings as fact. This exact example was demonstrated in the SPIIN trial in a cohort of oncologists.1 Therefore, educating cardiologists about spin includes educating them to fight optimism bias,16 which may prevent them from indiscriminately favoring new interventions. Further, educating peer reviewers is essential, given the understanding that peer reviewers are not adequately trained to detect spin.17
Moreover, multiple investigations have shown that studies producing positive or statistically significant results are more likely to be published, cited, and accepted by highly ranked journals.18-21 This overrepresentation of positive findings in the published literature is known as positive publication bias, which can lead to overestimation of the effectiveness of the intervention when pooling the results of statistically-significant studies alone into a meta-analysis.21 To mitigate this bias and increase the number of studies with nonsignificant findings, it is vital that the medical research community focus on publishing not only studies with significant results but also studies with nonsignificant findings.
We believe the following 3 steps may help decrease the negative results that stem from positive publication bias and spin. First, prospective study registration is a promising solution to mitigate positive publication bias.21 Trial registries, such as ClinicalTrials.gov, catalog all prospective trials that meet particular criteria, as well as observational studies. There are many registries worldwide, and these registries have made searching for unpublished studies easier. Further, these registries, in some cases, require that authors deposit study results within 1 year of completion.22 Second, we recommend that funding agencies require authors to publish studies they fund, regardless of the nature of the findings. Some funding agencies, such as the Wellcome Trust, have created journals to ensure that the studies they fund are able to be published.23 Lastly, journals should consider incentivizing the publication of nonsignificant results in their policies or featuring special sections of the journal that focus on the publication of nonsignificant results. We believe these 3 recommendations will help increase the quantity of nonsignificant studies published and decrease the bias that may arise from having only positive results recognized in the literature.
Our study has several limitations. While we used double-blinded data extraction that was preceded by rigorous training to ensure the accuracy of data extraction and analysis, the assessment of spin is necessarily subjective to a degree, and our results may not be generalizable because of our journal and date parameters. Further, we were limited because some articles did not properly define their primary endpoints. Continued assessments of spin in the cardiology literature, both in formal cross-sectional analyses like the present study and informal public review, are required to identify and address spin in the cardiology literature.
Conclusion
Our study found that spin is prevalent in the abstracts of included RCTs. The implications of spin include, but are not limited to, exaggerated beliefs in the efficacy of pharmacological treatments. Solutions to spin include educating cardiologists to detect it, educating authors to mitigate spin in their published trials, and incentivizing the publication of nonsignificant results.
Author Contributions
Mr. Cooper and Dr. Khattab provided substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; Mr. Roberts and Dr. Vassar drafted the article or revised it critically for important intellectual content; Mr. Roberts and Dr. Vassar gave final approval of the version of the article to be published; and all authors agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
References
1. BoutronI, AltmanDG, HopewellS, et al.. Impact of spin in the abstracts of articles reporting results of randomized controlled trials in the field of cancer: the SPIIN randomized controlled trial. J Clin Oncol.2014;32(36):4120-6. doi:10.1200/JCO.2014.56.7503.Search in Google Scholar PubMed
2. BoutronI, DuttonS, RavaudP, et al.. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA. 2010;303(20):2058-2064. doi:10.1001/jama.2010.651Search in Google Scholar PubMed
3. SaintS, ChristakisDA, SahaS, et al.. Journal reading habits of internists. J Gen Intern Med.2000;15(12):881-884. doi:10.1046/j.1525-1497.2000.00202.xSearch in Google Scholar
4. WilsonJR. Rhetorical strategies used in the reporting of implantable defibrillator primary prevention trials. Am J Cardiol.2011;107(12):1806-1811. doi:10.1016/j.amjcard.2011.02.320Search in Google Scholar PubMed
5. ColeWayant MV. Central protocol for assessing spin in the biomedical literature. June 15, 2018. Accessed September 9, 2020. https://osf.io/eu5t2/.Search in Google Scholar
6. Google Scholar Metrics. Accessed September 1, 2020. https://scholar.google.com/intl/en/scholar/metrics.html#metricsSearch in Google Scholar
7. OuzzaniM, HammadyH, FedorowiczZ, et al.. Rayyan—a web and mobile app for systematic reviews. Syst Rev.2016;5:210.10.1186/s13643-016-0384-4Search in Google Scholar PubMed PubMed Central
8. ChiuK, GrundyQ, BeroL. ‘Spin’ in published biomedical literature: a methodological systematic review. PLoS Biol.2017;15(9):e2002173. doi:10.1371/journal.pbio.2002173Search in Google Scholar PubMed PubMed Central
9. JellisonS, RobertsW, BowersA, et al.. Evaluation of spin in abstracts of papers in psychiatry and psychology journals. BMJ Evid Based Med. Epub ahead of print5 August 2019. doi:10.1136/bmjebm-2019-111176.Search in Google Scholar PubMed
10. CooperCM, GrayHM, RossAE, et al.. Evaluation of spin in the abstracts of otolaryngology randomized controlled trials. Laryngoscope.2018;2036-2040. doi:10.1002/lary.27750Search in Google Scholar PubMed
11. BarryHC, EbellMH, ShaughnessyAF, et al.. Family physicians’ use of medical abstracts to guide decision making: style or substance?J Am Board Fam Pract.2001;14(6):437-442.Search in Google Scholar
12. MarceloA, GavinoA, Isip-TanIT, et al.. A comparison of the accuracy of clinical decisions based on full-text articles and on journal abstracts alone: a study among residents in a tertiary care hospital. Evid Based Med.2013;18(2):48-53. doi:10.1136/eb-2012-100537Search in Google Scholar PubMed PubMed Central
13. YavchitzA, BoutronI, BafetaA, et al.. Misrepresentation of randomized controlled trials in press releases and news coverage: a cohort study. PLOS Med.2012. doi:10.1371/journal.pmed.1001308Search in Google Scholar PubMed PubMed Central
14. PhillipsDP, KanterEJ, BednarczykB, et al.. Importance of the lay press in the transmission of medical knowledge to the scientific community. N Engl J Med.1991;325(16):1180-1183. doi:10.1056/NEJM199110173251620Search in Google Scholar PubMed
15. NielsenSF, NordestgaardBG. Negative statin-related news stories decrease statin persistence and increase myocardial infarction and cardiovascular mortality: a nationwide prospective cohort study. Eur Heart J.2016;37(11):908-916. doi:10.1093/eurheartj/ehv641Search in Google Scholar PubMed
16. DjulbegovicB, KumarA, MagazinA, et al.. Optimism bias leads to inconclusive results-an empirical study. J Clin Epidemiol.2011;64(6):583-593. doi:10.1016/j.jclinepi.2010.09.007Search in Google Scholar PubMed PubMed Central
17. LazarusC, HaneefR, RavaudP, et al.. Peer reviewers identified spin in manuscripts of nonrandomized studies assessing therapeutic interventions, but their impact on spin in abstract conclusions was limited. J Clin Epidemiol.2016;77:44-51. doi:10.1016/j.jclinepi.2016.04.012Search in Google Scholar PubMed
18. DuyxB, UrlingsMJE, SwaenGMH, et al.. Scientific citations favor positive results: a systematic review and meta-analysis. J Clin Epidemiol.2017;88:92-101. doi:10.1016.j.jclinepi.2017.06.00210.1016/j.jclinepi.2017.06.002Search in Google Scholar PubMed
19. FanelliD. Negative results are disappearing from most disciplines and countries. Scientometrics.2012;90:891-904. https://doi.org/10.1007/s11192-011-0494-710.1007/s11192-011-0494-7Search in Google Scholar
20. FanelliD. Do pressures to publish increase scientists’ bias? An empirical support from US States Data. PLoS One.2010;5(4):e10271. Published 2010 Apr 21. doi:10.1371/journal.pone.001027110.1371/journal.pone.0010271Search in Google Scholar PubMed PubMed Central
21. MontoriVM, SmiejaM, GuyattGH. Publication bias: a brief review for clinicians. Mayo Clin Proc.2000;75(12):1284-1288. doi:10.4065/75.12.1284Search in Google Scholar PubMed
22. CollinsF. Clinical Trials Registration and Results Information Submission. September 8, 2016. Accessed March 26, 2020. https://www.federalregister.gov/documents/2016/09/21/2016-22129/clinical-trials-registration-and-results-information-submissionSearch in Google Scholar
23. Wellcome Open Research. Accessed March 26, 2020. https://wellcomeopenresearch.org/Search in Google Scholar
© 2020 American Osteopathic Association
