A comparison of quality of abstracts of systematic reviews including meta-analysis of randomized controlled trials in high-impact general medicine journals before and after the publication of PRISMA extension for abstracts: a systematic review and meta-analysis

We identified abstracts of systematic reviews published in 2012, 2014, and 2015 in high-impact general medicine journals. We assessed the reporting of these abstracts according to PRISMA-A recommendations [14]. We compared reporting from 2012 (before PRISMA-A) with reporting in 2014 and 2015 (after PRISMA-A). The present review was not registered. The PRISMA guidelines served as the template for reporting the present review [16]. The PRISMA checklist can be found in the Additional file 1.

Based on impact factor published in 2015 by Thomson Reuters [17], the top nine high-impact general medical journals were selected for this study, namely: New England Journal of Medicine (NEJM), The Lancet, Journal of American Medical Association (JAMA), Annals of Internal Medicine, British Medical Journal (BMJ), Archives of Internal Medicine, PLOS Medicine, JAMA Internal Medicine, BMC Medicine, and Mayo Clinic Proceedings. In the PRISMA website, NEJM, Annals of Internal Medicine, Archives of Internal Medicine, and Mayo Clinic Proceedings are not cited as endorsers of PRISMA guidelines by contrast to the other journals which are PRISMA endorsers [18]. Journals were not excluded on the basis of their lack of endorsement of the PRISMA statement. We conducted a PubMed search of all systematic reviews and meta-analyses published in years 2012, 2014, and 2015. Our search strategy included “meta-analysis” as publication type, journal names, and limits were set for the specific time periods of interest (2012/01/01 to 2012/12/31, 2014/01/01 to 2014/12/31, and 2015/01/01 to 2015/12/31). The search strategy is presented in the Additional file 2.

Two reviewers (JJRB and LNU) independently selected abstracts of systematic reviews of RCTs including meta-analyses. A “yes” or “no” answer was assigned to each item indicating whether the authors had reported it or not. Data extraction was independently performed by two reviewers (JJRB and LNU) in compliance with recommendations of the evaluation of PRISMA-A guidelines [14]. This was done for the 12 items recommended by the PRISMA-A guidelines. Agreement was measured using the Kappa statistic [19]. Disagreements were resolved by consensus after discussion between authors.

In addition to the PRISMA-A items, we collected the journal name, number of authors, type of abstract format (Introduction, Methods, Results, and Discussion [IMRAD]; 8-headings [objective, design, setting, participants, intervention, main outcome measured, results, and conclusions]), PRISMA endorser journal (yes or no), and actual observed abstract word count (<300 versus ≥300).

The primary outcome was the number of items reported among the 12 recommended items of the PRISMA-A guidelines; the secondary one was the proportion of abstracts reporting each of these 12 recommended items.

We expressed the number of items for each year as mean (SD) and estimated the unadjusted and adjusted differences using the independent two-sample Student t test and generalized estimation equations (GEE), respectively [20]. The mean differences and adjusted means were reported with their 95 % confidence intervals (95 % CI) and p values. For continuous variables (number of items reported, for example), we assumed a linear distribution. We compared compliance with the 12 items of the PRISMA-A for years 2012 versus 2014 and 2015 using individual chi-squared test or Fischer’s exact test where necessary. Unadjusted analysis was followed by an adjusted analysis using GEE. For binary outcomes, we assumed a binomial distribution and an unstructured correlation matrix. We reported measures of association with odds ratios (OR) for univariable analyses and adjusted odds ratios (aOR) for multivariable analyses together with their 95 % confidence intervals (CI) and p values. Factors associated with overall better reporting were investigated using GEE and assuming Poisson distribution for number of items reported. The magnitude of association between better reporting and investigated factors was measured with adjusted incidence rate ratio (aIRR) alongside its 95 % CI. Additionally, we conducted sensitive analyses to determine factors associated with a better quality of reporting of the “Methods” and “Results” sections of abstracts for years 2014 and 2015.

For GEE, adjustments were made for actual observed abstract word count (<300 versus ≥300), PRISMA endorser journal (yes versus no), abstract format (IMRAD versus 8-headings), publication on behalf of a group (yes versus no), number of authors (≤6 versus >6), with journal as a grouping factor to adjust for potential clustering or similarity in articles published in the same journal. Evidence against the null hypothesis was considered for a two-tailed p value of <0.05. The 300-word cut-off was chosen as it has been shown, like for other abstract reporting guidelines such as the Consolidated Standards of Reporting Trials (CONSORT) extension for abstracts, that a word count around 300 is sufficient to fill all items [20]. The adjustment was done for abstract format because there are previous studies reporting relationship between abstract format and quality reporting [21, 22]. The adjustment was also done for number of authors and publication on behalf a group because it was reported an association between higher quality of work and increased number of collaborators [23, 24], though some other studies did not end up with the same finding [25, 26].

Our search yielded 505 articles of which 183 were published in 2012, 192 in 2014, and 130 in 2015. Three hundred and one abstracts did not meet our inclusion criteria, hence were excluded because most of them not included RCTs in the meta-analysis (n = 280). On the whole, we included 204 abstracts: 84 from 2012, 59 from 2014, and 61 from 2015 as shown in the flow diagram (Fig. 1). Agreement between reviewers on all PRISMA-A items was high (Kappa = 0.79, p < 0.001). Additional file 3 lists all abstracts included in this review.

As concerns the journals in which the abstracts were published, 67 (32.8 %) were published in BMJ (Table 1). The 8-headings format was used in 132 (64.7 %) abstracts. Among all the journals, 163 (79.9) were published with an actual observed abstract word count ≥300, 19 (9.3 %) abstracts reported publications on behalf of a group in authorship, and 150 abstracts (73.5 %) were from journals which endorsed PRISMA. Four of the 10 journals were PRISMA endorsers. The mean number of authors per article was 8.5 (SD 5.4) (Table 1).

The mean number of items reported in 2014 (mean = 6.8; SD = 1.6) did not statistically differ from that reported in 2012 (mean = 6.7; SD = 1.6), mean difference (MD): 0.1; 95 % CI −0.4; 0.7; p = 0.694. There was no statistically significant difference after adjusting for covariates among which actual observed abstract word count, PRISMA endorsement, abstract format, publication on behalf of a group, and number of authors (MD −0.1, 95 % CI −0.6; 0.4; p = 1.0). The mean number of items reported in 2015 (mean = 7.5; SD = 1.6) differed significantly from that reported in 2012: mean difference (MD): 0.8; 95 % CI 0.2; 1.3; p = 0.007. This difference remained statistically significant after adjustment for covariates (MD 0.9, 95 % CI 0.4; 1.3; p = 0.002) (Table 2). An exploratory comparison between abstracts released in 2015 and those released in 2014 with regard to the overall quality of reporting showed a statistically significant difference (MD 0.63, 95 % CI 0.05; 1.21; p = 0.034).

Seven items were reported in more than 50 % of the abstracts: title, objectives, eligibility criteria, included studies, synthesis of results, description of effect, and interpretation. Four items were reported in less than 50 % of the abstracts: information sources, strengths and limitations of evidence, funding, and registration. Reporting of risk of bias varied, around 50 % throughout years (49.2–67.8 %) (Table 3).

In univariable analysis, there was a statistically significant improvement in abstract reporting in 2014 compared to 2012 only for “risk of bias” (crude OR [cOR] = 2.0; 95 % CI 1.003; 4.0). After adjustment, “risk of bias” became insignificant while reporting of “strengths and limitations of evidence” (adjusted OR [aOR] = 0.23; 95 % CI 0.07–0.71) and “Funding” (aOR = 0.25; 95 % CI 0.07–0.81) became statistically associated with a lower quality in 2014 compared to 2012 (Table 4).

(cOR not estimable, p < 0.001), and for “registration” (cOR = 12.2; 95 % CI 2.7–56.1). The quality of abstract decreased from 2012 to 2015 concerning “strength and limitations of evidence” (cOR = 0.38; 95 % CI 0.18–0.82). After adjustment, “synthesis and results” (aOR not estimable, p < 0.001), “description of effect” (aOR = 2.7; 95 % CI 1.2–6.1), “interpretation” (aOR not estimable, p < 0.001), and “registration” (aOR = 10.8; 95 % CI 2.3–49.6) were statistically associated with a better quality of reporting in 2015 compared to 2012. “Strengths and limitations of evidence” was statistically associated with lower reporting quality in 2015 compared to 2012 (Table 4).

In 2014, factors statistically associated with overall better reporting were structuring abstracts in the 8-headings format compared to the IMRAD format (aIRR 1.26; 95 % CI 1.02–1.56) and abstract word count <300 (aIRR 1.20; 95 % CI 1.09–1.35). The sole factor statistically associated with a better reporting of the “Methods” section was abstract word count <300 (aIRR 1.43; 95 % CI 1.16–1.72). None of the researched factors was statistically associated with a better quality of reporting of the “Results” section (Table 5).

In 2015, the sole factor statistically associated with an overall better reporting was abstract word count <300 (aIRR 1.20; 95 % CI 1.09–1.35). Factors statistically associated with a better reporting of the “Methods” section was abstract word count <300 (aIRR 1.43; 95 % CI 1.16–1.72), structuring abstract in the 8-headings format compared to the IMRAD format (aIRR 1.33; 95 % CI 1.08–1.65), and not publishing on behalf of a group (aIRR 1.37; 95 % CI 1.04–1.82). No factor was statistically associated with a better reporting of the “Results” section (Table 6).