The online version of this article (doi:10.1186/1471-2288-14-45) contains supplementary material, which is available to authorized users.
Carson Ka-Lok Lo, Dominik Mertz contributed equally to this work.
The authors declare that they have no competing interests.
DM and ML designed the study. CKL drafted the survey, all authors contributed to revising the survey and provided a final version to be sent to authors of interest. DM responded to all inquiries from authors regarding the survey. CKL contributed to data collection, data analysis, and interpretation. DM and ML provided feedback on interpretation of data. CKL drafted the manuscript. All authors critically revised the manuscript and provided final approval of the version to be published.
Lack of appropriate reporting of methodological details has previously been shown to distort risk of bias assessments in randomized controlled trials. The same might be true for observational studies. The goal of this study was to compare the Newcastle-Ottawa Scale (NOS) assessment for risk of bias between reviewers and authors of cohort studies included in a published systematic review on risk factors for severe outcomes in patients infected with influenza.
Cohort studies included in the systematic review and published between 2008–2011 were included. The corresponding or first authors completed a survey covering all NOS items. Results were compared with the NOS assessment applied by reviewers of the systematic review. Inter-rater reliability was calculated using kappa (K) statistics.
Authors of 65/182 (36%) studies completed the survey. The overall NOS score was significantly higher (p < 0.001) in the reviewers’ assessment (median = 6; interquartile range [IQR] 6–6) compared with those by authors (median = 5, IQR 4–6). Inter-rater reliability by item ranged from slight (K = 0.15, 95% confidence interval [CI] = −0.19, 0.48) to poor (K = −0.06, 95% CI = −0.22, 0.10). Reliability for the overall score was poor (K = −0.004, 95% CI = −0.11, 0.11).
Differences in assessment and low agreement between reviewers and authors suggest the need to contact authors for information not published in studies when applying the NOS in systematic reviews.
Additional file 1: Original NOS and survey questions for cohort studies. Original NOS (left) and survey questions (right) that were sent to authors. Criteria for awarding points in the survey were matched with the original NOS, as indicated by corresponding symbols in the document. (PDF 75 KB)
Additional file 2: Cohort studies included in analysis. References of all 65 cohort studies who completed the survey and included in data analysis. (PDF 125 KB)
Additional file 3: Raw data for NOS reviewer and author assessments. Risk of bias assessments of all 65 cohort studies by NOS reviewers and authors. Scores were tabulated and separated by NOS domain and item, with total scores included. (XLS 40 KB)
Cochrane handbook for systematic reviews of interventions version 5.1.0 [updated March 2011]. Edited by: Higgins JPT, Green S. [ http://www.cochrane-handbook.org]
Wells G, Shea B, O’Connell D, Peterson J, Welch V, Losos M, Tugwell P: The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. 2013, http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp,
Devereaux PJ, Choi PTL, El-Dika S, Bhandari M, Montori VM, Schünemann HJ, Garg AX, Busse JW, Heels-Ansdell D, Ghali WA, Manns BJ, Guyatt GH: An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods. J Clin Epidemiol. 2004, 57: 1232-1236. 10.1016/j.jclinepi.2004.03.017. CrossRefPubMed
Soares HP, Daniels S, Kumar A, Clarke M, Scott C, Swann S, Djulbegovic B: Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ. 2004, 328: 22-24. 10.1136/bmj.328.7430.22. CrossRefPubMedPubMedCentral
Mertz D, Kim TH, Johnstone J, Lam P-P, Science M, Kuster SP, Fadel SA, Tran D, Fernandez E, Bhatnagar N, Loeb M: Populations at risk for severe or complicated influenza illness: a systematic review and meta-analysis. BMJ. 2012, 347: f5061- CrossRef
Ben-David A: Comparison of classification accuracy using Cohen’s Weighted Kappa. Expert Syst Appl. 2008, 34: 825-832. 10.1016/j.eswa.2006.10.022. CrossRef
Wells G, Brodsky L, O’Connell D, Shea B, Henry D, Mayank S, Tugwell P: Evaluation of the Newcastle-Ottawa Scale (NOS): an assessment tool for evaluating the quality of non-randomized studies. XI Cochrane Colloquium: Evidence, Health Care and Culture. 2003, Barcelona, Spain, http://www.citeulike.org/user/SRCMethodsLibrary/article/12919189,
Strijbos J-W, Martens RL, Prins FJ, Jochems WMG: Content analysis: what are they talking about?. Comput Educ. 2006, 46: 29-48. 10.1016/j.compedu.2005.04.002. CrossRef
- Newcastle-Ottawa Scale: comparing reviewers’ to authors’ assessments
Carson Ka-Lok Lo
- BioMed Central
Neu im Fachgebiet AINS
Meistgelesene Bücher aus dem Fachgebiet AINS
Mail Icon II