A descriptive analysis of non-Cochrane child-relevant systematic reviews published in 2014

We conducted a cross-sectional descriptive study of all SRs on any child-relevant topic published outside of the Cochrane Library over 1 year. Ethics approval was not required because there were no human participants.

English language child-relevant SRs of quantitative primary research published outside of the Cochrane Library in 2014 were eligible. We used Cochrane [10] standards to define SRs for inclusion, with some flexibility (e.g., we did not require a quality appraisal) to account for the fact that many SRs published outside of the Cochrane library might not meet their stringent criteria nor adhere to published reporting standards such as the PRISMA statement (Preferred Reporting Items for Systematic reviews and Meta-Analyses). [11, 22] Thus, we considered SRs to be those publications that reported an explicitly defined methodology, including: [10, 11, 22] (a) a comprehensive search (i.e., at least two databases or one database and a supplementary method such as hand-searching); (b) a description of study selection, including the number of records identified via the search and primary studies retained for the review; (c) predefined eligibility criteria; (d) a description of the data extraction methodology and/or items extracted; and (e) a summary of findings. We excluded Cochrane SRs (including duplicate or summary publications published outside the Cochrane Library), theses and dissertations, and all other types of reviews (i.e., rapid reviews, scoping reviews, iterative reviews, reviews of qualitative studies, mixed methods reviews, overviews of reviews).

We defined ‘child-relevant’ SRs as those that intended to include children (0 to 18 years) or presented at least one outcome pertaining to children. [20, 21] This included interventions on adults that were intended to improve the health of children (e.g., smoking cessation programs for post-partum mothers [23]), and reviews including both children and adults, even when child-specific subgroup analyses were not conducted. We included SRs on neonates, those that reported on outcomes measured in adults as a result of exposures occurring in childhood (e.g., post-traumatic stress resulting from child abuse [24]) and those relating to pregnancy and breastfeeding with child-relevant outcomes (e.g., birth weight, [25] congenital malformations, [26] outcomes in the child resulting from breastfeeding [27]). We excluded SRs reporting only fetal outcomes, preterm delivery or stillbirth.

A research librarian developed a search strategy to locate child-relevant SRs that included subject headings related to the pediatric population, and SRs (Additional file 1). Searches were undertaken on August 17, 2015 in Ovid MEDLINE(R) In-Process & Other Non-Indexed Citations and Ovid Medline(R) 1946 to present, CINAHL via EBSCOhost, the Web of Science Core Collection, and PubMed. We applied a methodological search filter to restrict the search results to SRs. [28] The searches were also restricted to English language SRs in humans published from January 1 to December 31, 2014.

We employed a two-stage selection process. First, one reviewer screened the titles and abstracts of all records and classified them as ‘include’, ‘exclude’ or ‘unsure’. A second reviewer screened all ‘excludes’. We then retrieved the full text of any record categorized as ‘include’ or ‘unsure’ by either reviewer. We followed the same process for full text screening. Disagreements were resolved by discussion or the involvement of a third reviewer.

One reviewer developed a data extraction form in Microsoft Office Excel (v. 2016, Microsoft Corporation, Redmond, WA). Three reviewers independently pilot tested the form on a sample of 5 to 10 SRs. Then, a single reviewer independently extracted data from the SRs. The three reviewers met regularly to resolve uncertainties.

We extracted three categories of data, similar to previous descriptive analyses of child-relevant Cochrane SRs, [20, 21] to optimize comparability: 1) general SR characteristics i.e., country of the corresponding author and income classification (World Bank [29]), type of journal, type of review question (i.e., therapeutic [treatment and prevention], epidemiological [prevalence, association between exposure and outcome, genetic association reviews], diagnostic/prognostic [accuracy of diagnostic tests, prognostic factors, clinical prediction rules], or other [cost-effectiveness, psychometric properties of measurement tools, others that do not fit any group]), [9] topic of the review based on existing Cochrane Review Groups (CRGs, to allow comparison in the quantity of evidence by topic area), [30] whether the SR was an update, sources of funding, mention of published or registered review protocol and/or registration within a SR registry; 2) characteristics of the included primary studies i.e., primary study designs sought and included, the number of reports of studies and number of human participants included, and the type of participants included; and 3) methodological approaches i.e., statement of an objective and primary outcome, quality appraisal of the included studies, assessment of certainty of body of evidence (defined as “extent of our confidence that the estimates of effect are correct” [31]), and synthesis method. Additional file 2 shows detailed information about the data extraction items. We extracted the data as reported in the SRs, and did not consult authors, search for published protocols or registration, or use other methods to acquire additional information.

We performed univariate analyses in Excel to describe the general characteristics of the SRs, their included studies and methodological approaches. We calculated counts and frequencies for categorical data, and medians and interquartile ranges for continuous data. We presented the findings narratively.

We retrieved 5213 unique records via the search and excluded 2070 during title and abstract screening. We inspected the full text of 3143 records and excluded 1545 (Fig. 1). Most full text exclusions were related to publications not meeting our definition of a SR (n = 1258/1545, 81.4%). The final sample included 1598 child-relevant SRs containing a median (IQR) 19 (11 to 33) reports of studies and 2134 (648 to 13,079) participants.

Tables 1, 2 and 3 show the SR, included primary studies, and methodological characteristics of the sample, overall and by type of review question. The SRs originated almost exclusively from high (n = 1247, 78.0%) and upper-middle (n = 317, 19.8%) income countries. The USA (n = 384, 24.0%), China (n = 201, 12.6%), the UK (n = 201, 12.6%), Canada (n = 150, 9.4%), and Australia (n = 136, 8.5%) were the top child-relevant SR-producing countries.

The SRs spanned 47 of the 53 CRGs and the 8 most populated review groups contained 53.2% (n = 849) of the SRs: Developmental, Psychosocial and Learning Problems (n = 216, 13.5%); Metabolic and Endocrine Disorders (n = 132, 8.3%); Pregnancy and Childbirth (n = 124, 7.8%); Common Mental Disorders (n = 110, 6.9%); Airways (n = 70, 4.4%); Anaesthesia, Critical and Emergency Care (n = 69, 4.3%); Public Health (n = 67, 4.2%); and Infectious Diseases (n = 61, 3.8%). Some notable CRGs containing very few SRs were Skin (n = 15, 0.9%), Tobacco Addiction (n = 15, 0.9%), Back and Neck (n = 14, 0.9%), Neonatal (n = 14, 0.9%; neonatal reviews of RCTs only), Eyes and Vision (n = 12, 0.8%), Haematological Malignancies (n = 9, 0.6%), Neuromuscular (n = 7, 0.4%), and Urology (n = 3, 0.2%).

Most often, the SRs synthesized therapeutic (n = 753, 47.1%) or epidemiologic (n = 701, 43.8%) evidence. Although 39.3% (n = 628) of the SRs included evidence specific to children only, fewer were published in pediatric-specific journals (n = 283, 17.7%). Approximately half of SRs (n = 807, 50.5%) declared at least one external source of support; 15.5% (n = 248) had no external funding. The most common sources of support were governments (n = 557, 34.9%), academic or research institutes (n = 228, 14.3%), and foundations (n = 193, 12.1%).

Only three SRs (< 0.01%) included all of the methodological and reporting characteristics that we assessed. Few mentioned a published or registered protocol (n = 246, 15.4%) or SR registration (n = 111, 6.9%). One-third neglected to mention sources of support (n = 543, 34.0%) and more than half did not indicate the total number of included human participants (n = 902, 56.6%). Most (n = 1224, 76.6%) did not specify one or more primary outcome(s). In 9.9% (n = 158) of cases, eligible primary study designs were not mentioned, and in 9.2% (n = 147) the type(s) included in the SR was not discernable. Few SRs were identified as updates (n = 32, 2.0%). Some were labelled as updates by their authors, but appeared to be new.

Authors rarely searched only for RCTs (n = 203, 12.7%). Most commonly, both RCTs and other primary study designs were eligible (n = 765, 47.9%). This was reflected in the included primary studies, where most SRs relied solely on evidence from non-RCTs (n = 796, 49.8%) or a mix of designs (n = 420, 26.3%). Less than two-thirds of SRs (n = 953, 59.6%) included an assessment of the quality of the included studies and very few assessed the certainty of the body of evidence (n = 102, 6.4%; using Grading of Recommendations Assessment, Development and Evaluation (GRADE) (n = 72, 4.5%) or another method (n = 30, 1.9%)). The synthesis method used in SRs was roughly equally split between narrative (n = 741, 46.4%) and statistical (n = 857, 53.6%). There were 5 network meta-analyses and 12 individual patient data meta-analyses.

Epidemiological SRs included the largest number of primary studies (median (IQR), 23 (13 to 40)) and human participants (11,038 (2098 to 63,897)). Compared to the overall sample, a larger proportion of diagnostic/prognostic SRs were published in Asia (n = 25/91, 27.5%), and therapeutic SRs in North America (n = 287/753, 38.1%). The Injuries CRG was the most strongly represented group among diagnostic/prognostic SRs, while the Developmental, Psychosocial, and Learning Problems CRG was predominant across all other review types.

Compared to the overall sample, epidemiological SRs more often included mixed samples of children and adults (n = 384/701, 54.8%) and less often included only children (n = 212/701, 30.2%), but the type of journal in which they were published did not differ. In contrast, the ‘other’ SRs were more often published in pediatric journals (n = 16/53, 30.1%), though the sample of these was small. The highest proportion of unfunded SRs was in the therapeutic group (n = 130/753, 17.3%). Industry funding was most common among the diagnostic/prognostic (n = 5/91, 5.5%) and ‘other’ (n = 3/53, 5.7%) SRs.

Reporting of the items of interest that we collected was most complete for the therapeutic SRs and least for the epidemiological and ‘other’ SRs. The therapeutic SRs most often indicated the type of primary studies sought (n = 727/753, 96.5%) and included (n = 734/753, 97.5%), reported the total number of human participants (n = 370/753, 49.4%), and identified a primary outcome (n = 281/753, 37.3%). The epidemiological SRs least often identified a published or registered SR protocol (n = 90/701, 12.8%) or registration in a SR registry (n = 35/701, 5.0%), and most often failed to indicate the type of primary studies sought (n = 114/701, 16.3%) and included (n = 111/701, 15.8%). The total number of participants (n = 273/701, 38.9%) and primary outcome (n = 74/701, 10.6%) were relatively infrequently reported among epidemiologic SRs, though the ‘other’ SRs also performed poorly in this respect.

All but one of the SRs seeking to include only RCTs was therapeutic; the epidemiologic, diagnostic/prognostic, and ‘other’ SRs more commonly sought only non-RCTs (49.4%, 52.7%, 54.7%, respectively). Non-RCTs were eligible for inclusion among 69.7% (n = 525/753) of the therapeutic SRs. The quality of the included studies was least frequently assessed in the epidemiologic SRs (n = 325/701, 46.4%), and most frequently in therapeutic (n = 532/753, 70.7%) and diagnostic/prognostic (n = 63/91, 69.2%) SRs. Certainty of the body of evidence was most often assessed in therapeutic SRs (n = 75/753, 10.0%), and rarely in epidemiologic (n = 20/701, 2.8%) or diagnostic/prognostic (n = 3/91, 3.3%) SRs.

We have identified key areas in need of improvement in the methodology and reporting of child-relevant SRs. This assessment was based on several reporting standards of interest, but we did not formally appraise the SRs against specific guidance for methods or reporting. In the future, an in-depth characterization of the included SRs against established guidance (e.g., PRISMA, [11, 22] AMSTAR 2 [41]) would add value. One limitation of our sample is that it included only English-language SRs of quantitative data published in 1 year. Other types of reviews fell outside the scope, and SR topics will vary over time. Related to the inconsistent, and overall poor reporting, it was often difficult to extract key information from the SRs. Also, one reviewer extracted data from the SRs, which increased the risk of error compared to duplicate extraction. [42] We mitigated these risks by pilot testing the data extraction form, meeting regularly to address concerns, and re-evaluating data items with poor agreement between reviewers. As we did not seek additional information about the SRs, it was not possible to separate methodological from reporting deficits.