Background
Methods
Scope
Inclusion criteria
- (1) methodological studies that set out to investigate data dredging/p-hacking, outcome reporting bias or publication bias by one or more of: a) tracking a cohort of studies from inception or from a pre-publication stage such as conference presentation to publication (or not); b) surveying researchers about their experiences related to research publication; c) investigating statistical techniques to prevent, detect or mitigate the above biases;
- (2) systematic reviews of substantive HSR topics that provided empirical evidence concerning publication and related biases. Such evidence could take various forms such as comparing findings in published vs. grey literature; statistical analyses (e.g. funnel plots and Egger’s test); and assessment of selective outcome reporting within individual studies included in the reviews.
Exclusion criteria
Search strategy
Data extraction
Methodological studies
Systematic reviews of substantive topics of HSDR
Risk of bias assessment
Data synthesis and presentation
Results
Literature search and selection
Methodological studies setting out to investigate publication and related biases
Study (HSDR Topic) | Objectives | Methods | Key Findings | Limitations |
---|---|---|---|---|
Ammenwerth, 2007 [16] (Health Informatics) | To determine: - what percentage of IT evaluation studies are not published in international journals or proceedings - what are typical reasons for not publishing the results of an IT evaluation study | Written, e-mail-based survey of academics. Survey sample included members of several mailing lists and first authors of IT evaluation papers that were published between 2001 and 2006 and Medline indexed | Only half of the evaluation studies reported by responders were published. Common reasons for non-publication included ‘not of interest for others’, ‘no time for writing’, ‘limited scientific quality’, ‘political and legal reasons’ and ‘only meant for internal use’ | Low response rate (19%, 136/722). Study could be influenced by sampling, response and recall bias |
Costa-Font, 2013 [18] (Health Policy) | To examine the winner’s curse phenomenon (studies needing to have more extreme results to be published in high-impact journals) and publication selection bias using quantitative findings on income and price elasticities as reported in health economics research | Funnel plot and multivariate analysis to examine the association between estimated effect sizes (and their statistical significance) and the impact factors of the journals in which they are published | Meta-regression analysis demonstrated that both publication bias (reflected by positive correlation between effect size and standard error) and the winner’s curse (reflected by an independent association between effect size and journal impact factor) influence the estimated income/price elasticity | Alternative explanations for the observed associations cannot be excluded. Literature in the field concerned are often reported in grey literature rather than academic journals |
Machan, 2006 [17] (Health Informatics) | To determine: - the percentage of evaluation studies describing positive, mixed or negative results -the possibility of statistical assessment of publication bias in health informatics - the quality of reviews and meta-analysis in health informatics with regard to publication bias | Descriptive analysis of random sample of 86 evaluation studies and planned to construct funnel plot Examined characteristics and quality of reviews and meta-analyses (n = 54) in medical informatics | For the primary studies, 69.8% had positive results, 14% negative and 16.3% unclassified For the reviews 36.6% had positive conclusion, 61.5% were inconclusive, and only one review came to a negative conclusion | Small number of comparable studies prevented the quantitative analysis of potentiation publication bias Proportion of studies/reviews with a positive conclusion may not be good indicators for the existence of publication bias |
Vawdrey, 2013 [19] (Health Informatics) | To measure the rate of non-publication and assess possible publication bias in clinical trials of electronic health records | Follow-up of health informatics trials registered in ClinicalTrials.gov (2000–2008) | Trials with positive results were more likely to be published compared with trials with null results (92% of trials with positive results [35/38] vs 75% of trials with neutral or negative results [12/16], but the difference was not statistically significant (p = 0.177a) | Sample size relatively small; no information could be obtained for 8 unpublished trials; completeness of trial registration and representativeness of registered uncertain |
Systematic reviews of HSR topics providing evidence on publication and related bias
HSR systematic reviews comparing published and grey/unpublished evidence
Study (HSDR Topic) | Topic | Methods of identifying grey literature/unpublished studies | Key Findings of comparison between published literature and grey literature/unpublished studies | Limitations |
---|---|---|---|---|
Maglione, 2002 [22] (Immunization Program) | Effectiveness of mass mailings to increase utilization of influenza vaccine among Medicare beneficiaries | Search of the Medicare Peer Review Organization Health Care Quality Improvement Project database | Six controlled trials were identified. Only one (earliest) trial reporting modest but statistically significant improvement in vaccination rate (2–8% depending on the format of the letter and location of the study) was published. Five subsequent trials which found smaller, clinically trivial improvement in vaccination rate of no more than 2% remained unpublished | The review only included a small number of trials identified from a single study registry and targeting a specific US population |
Batt,2004 [20] (Immunization program) | Costs, effects and cost-effectiveness of strategies to increase coverage of routine immunizations in low- and middle income countries | Hand searches in institutional documentation centres including WHO and USAID; interviews with 28 international experts; search of grey literature databases; searches of the internet, conference proceedings and webpages of pertinent organizations | Quality of data on effect and cost-effectiveness was similar between published and grey literature, but the quality of costing data was poorer in grey literature. Inclusion of grey literature doubled the quantity of available evidence. Interventions examined in the grey literature were more up to date, associated with more complex interventions aimed at health systems and better represented west Africa and the Middle East. Conclusions drawn from the two sets of literature therefore differed | Reviewed grey literature was mainly derived from international organizations with little coverage of national governments. Searches were limited to English keywords |
Fang, 2007 [21] (Organizational studies) | Relationships between organizational culture, organizational climate, and nurse’s job satisfaction and turnover | Extensive search of 35 databases, “footnote chasing”, and searching by author | Of the nine associations for which findings were compared between published articles and unpublished doctoral dissertations, significant differences were found for three of them: association between passive/defensive culture and job satisfaction; global climate and job satisfaction; and reward orientation climate and job satisfaction. All the differences were related to magnitude rather than direction of the estimated association | Grey literature was limited to doctoral dissertations. The number of studies was very small for some of the comparisons; in some cases only one published or unpublished study was available |