Introduction
Methods
Development of a classification of spin
Prevalence of spin in abstracts
Search strategy
Study identification
Data extraction
Level of spin in abstract conclusions
Statistical analysis
Results
Selected articles
Characteristics | n = 128 (%) |
---|---|
Study design | |
Prospective | 42 (33 %) |
Before–after | 40 (31 %) |
Historical cohort | 39 (30 %) |
Cross-sectional | 6 (5 %) |
Other | 1 (1 %) |
Therapeutic intervention | |
Drug | 51 (40 %) |
Surgery | 14 (11 %) |
Device or equipment | 14 (11 %) |
Behavioral intervention or participative | 20 (16 %) |
Therapeutic strategy | 29 (23 %) |
Comparator | |
Placebo or attention control intervention | 4 (3 %) |
Active treatment | 55 (43 %) |
Usual care or no treatment | 25 (20 %) |
No comparator or Unclear | 44 (34 %) |
Sample size, median [Q1–Q3], (range) | 129.5 [50.75–457.5], (5–238600) |
Funding source | |
Profit | 28 (22 %) |
Not for profit | 54 (42 %) |
Both | 5 (4 %) |
Not reported or unclear | 41 (32 %) |
Registration | |
Yes | 39 (30 %) |
No or not reported | 89 (70 %) |
Classification of spin
Category of spin | Strategy used | Definition | Example |
---|---|---|---|
Misleading reporting | Not reporting adverse events or lack of focus on harm | Results are reported without warnings on important or relevant safety issue. | “This trial showed that overnight switch from oral ropinirole to transdermal rotigotine, with a dose conversion ratio of 1.5:1, was well tolerated in Korean patients with no apparent loss of efficacy.” 45/116 experienced adverse events and 13 stopped treatments because of adverse events. |
Selective reporting | Only a subset of the original outcomes or analysis planned in a study is fully reported. | “The results suggest that [Internet-based cognitive therapy] ICBT with therapist support has the potential to reduce [obsessive-compulsive disorders] OCD symptoms, depressive symptoms and general functioning”. No report of the lack of improvement in quality of life. | |
Misleading description of study design | Study design is presented as more robust than it is actually. | “Based on this prospective case control study tranexamic acid seems not to have a benefit in posterior lumbar spine surgery.” It was a retrospective study involving 97 patients and nothing was prospective in this study. | |
Use of linguistic spin | Any word or expression emphasizing the beneficial effect of the therapeutic intervention | “Both operative methods could be performed safely in the early stages after the introduction of surgery and could consistently obtain excellent surgical performance.” | |
No consideration of the limitations | Important limitations are not taken into account in the interpretation of the results. | “The TABADO program, targeting teenagers in vocational schools, was effective in producing a higher 12-month abstinence rate among all smokers in the intervention group.” No adjustment was made on major confounding variables. | |
Selective citation of other studies | Only previous studies concordant with the current study findings are acknowledged or other important studies in the field are not reported. | “It would be interesting to know its efficacy and safety in correcting high myopic astigmatism and how it changes the shape of the cornea.” Several publications already exist in this field. | |
Inadequate interpretation | Claim an effect for non-statistically significant results | Therapeutic intervention is presented as effective despite a non-statistically significant result. | “The use of [Automated CardioPulmonary Resuscitation] A-CPR resulted in a higher rate of survival to hospital compared with [Conventional CardioPulmonary Resuscitation] CPR” in a restrospective study involving 66 patients where the propensity score adjusted Odds Ratio was 1.69 [0.79; 3.63]. |
Claim an equivalence for non-statistically significant results despite a wide confidence interval | Therapeutic intervention and comparator are presented as equivalent when a comparison test is not statistically significant with a large confidence interval. | “The authors concluded that: 1) mortality during follow-up was statistically similar for both groups; (…)” In this retrospective cohort study involving 352 patients, the survival rate at 20 year was 69.3 % with mechanical mitral valve substitutes and 56.6 % with biological substitutes. The hazard ratio was not statistically significant but had a wide 95 % confidence interval (HR = 1.21 [0.79; 1.86], p = 0.386). | |
Ruling out safety for non-statistically significant results | Therapeutic intervention is presented as safe based on a non-statistically significant comparison test, despite a large confidence interval. | “Long-term treatment with esomeprazole (20 mg once daily) is well tolerated and efficacious” despite 16.29 % of patients (n = 22) experiencing adverse events judged to be possibly related to treatment with esomeprazole (mostly mild and transient). | |
Causal language or causal claim | Results are presented with a sentence implying a cause-and-effect link between the intervention and the outcome | “Treatment with oral valacyclovir as the sole antiviral therapy resulted in complete resolution of retinitis.” This was a before–after study involving 10 patients. | |
Claim of any significant difference despite lack of statistical test | Therapeutic intervention and comparator are compared despite no proper statistical test reported. | “pH and HCO3− significantly increased after hemodialysis sessions with both aspirin only and aspirin + dipyramidol, the increase of pH and HCO3−with aspirin only was significantly larger than aspirin + dipyramidol.” No statistical test was reported. | |
Focus on statistical significance instead of clinical relevance | Results are presented by their statistical significance without considering the clinical relevance of the effect size. | “While the [Clinical Global Impression-Schizophrenia] CGI-SCH overall score improved in both groups after switching, there was a significantly greater change in those who switched from olanzapine (difference of 0.29 points, p = 0.013)”. The CGI-SCH scale range from 0 to 7. | |
Inadequate extrapolation | Inadequate extrapolation to larger population, intervention or outcome | Results are generalized to another population, intervention or outcome than those of the study (such as surrogate outcomes) | “This intervention approach has the potential to impact on the progression of colorectal cancers and other cancers or chronic diseases.” The intervention focused on colorectal cancers only. |
Inadequate implication for clinical practice | Authors recommend the use of the therapeutic intervention for clinical practice. | “One should not hesitate to perform an osteotomy in difficult cases.” in a prospective cohort study involving 109 patients. |
-
Misleading reporting of the results was defined as incomplete reporting of the study results that could be misleading for the reader. This type of spin included 1) not reporting adverse events or lack of focus on harms (e.g., no warning on important safety issues), 2) selective reporting of outcomes favoring the beneficial effect of the experimental treatment (e.g., statistically significant results for efficacy outcomes or statistically non-significant results for harm outcomes), 3) misleading reporting of study design, 4) use of linguistic spin or “hype” (i.e., rhetorical manipulations to convince the readers of the beneficial effect of the treatment such as “excellent” results, “encouraging” outcomes, “a trend toward significance”), 5) no consideration of limitations, and 6) selective citation of other studies.
-
Inadequate interpretation of the results was defined as misleading interpretation of the study results overestimating the beneficial effect of the intervention. This type of spin included 1) claiming a beneficial effect of the intervention despite statistically non-significant results, 2) claiming an equivalent effect of the interventions for statistically non-significant results despite wide confidence interval, 3) claiming that the treatment is safe for statistically non-significant safety outcomes despite lack of power, 4) concluding a beneficial effect despite no comparison test performed, 5) interpretation of the results according to statistical significance (p-value) instead of clinical relevance, or 6) claiming a causal effect between the intervention being assessed and the outcome of interest despite a non-randomized design. Use of causal language was defined as any statement addressing the causal relationship of the intervention and outcomes with 1) modal auxiliary verbs, with the intervention as the subject and the outcome as a direct object (e.g., “Surgical experience could shorten the duration of TVT [tension-free vaginal tape] surgery.” [20]); 2) use of terms belonging to the semantic field of causal relationship (e.g., “effective”, “improve”, “enhance”); or 3) use of a tone inferring a strong result (e.g., “The results demonstrate” or “This study shows that”). We did not consider that causal language was used when authors stated only a co-occurrence between the intervention and the outcome (e.g., “subjects with symptomatic bipolar disorders who relapse frequently showed improvements in each of these areas after treatment with RLAI [risperidone long-acting injection]” [21]). We did not consider causal language as spin in studies using a propensity score or instrumental variables [22].
-
Inadequate extrapolation of the results was defined as an inappropriate generalization of the study result by inadequate 1) extrapolation from the population, interventions or outcome actually assessed in the study to a larger population, different interventions or outcomes, or 2) inadequate implications for clinical practice.
Prevalence of spin in abstracts
Spin categories | Presence of at least one example of spin in the abstract |
---|---|
n = 128 | |
At least one spin in the abstract | 107 (84.0) |
Misleading reporting | |
Not reporting adverse events or lack of focus on harm | 34 (26.6) |
Selective reporting | 15 (11.8) |
Misleading description of study designs | 1 (0.8) |
Use of linguistic spin or “hype” | 33 (25.8) |
Inadequate interpretation | |
Claim an effect for non-statistically significant results | 7 (5.5) |
Claim equivalence for non-statistically significant results | 13 (10.2) |
Ruling out safety when results are not statistically significant | 15 (11.8) |
Causal language or causal claim | 68 (53.1) |
Claim any difference despite no comparison test performed | 1 (0.8) |
Focus on statistical significance instead of clinical relevance | 1 (0.8) |
Inadequate extrapolation | |
Inadequate extrapolation to larger population, intervention or outcome | 11 (8.6) |
Inadequate implications for clinical practice | 25 (19.5) |
Other | 5 (3.9) |