Abstract

SYSTEMATIC REVIEW

An AMSTAR assessment of the methodological quality of systematic reviews of oral healthcare interventions published in the Journal of Applied Oral Science (JAOS)

Patrick Sequeira-Byron^I; Zbys Fedorowicz^II; Vanitha A. Jagannath^III; Mohammad Owaise Sharif^IV

^IBDS, Dr. Med. Dent., PGDipEBHC (Oxford), MSc, MSc Oberarzt, Department of Preventive Restorative and Pediatric Dentistry, University of Bern, Switzerland

^IIMSc DPH, BDS LDS RCS Eng., Director, The Bahrain Branch UK Cochrane Centre, Bahrain

^IIIMBBS, MD Pediatrics; Specialist Paediatrician, The Bahrain Branch UK Cochrane Centre, Bahrain

^IVBDS, MSc, MJDF RCS Eng., NIHR Academic Clinical Fellow, The University of Manchester, Manchester, United Kingdom

^{Corresponding address} Corresponding address: Dr. P. Sequeira-Byron Department of Preventive, Restorative and Pediatric Dentistry Freiburgstrasse 7 CH-3010 Bern - Switzerland Phone: +41 41 740 12 20 (B) e-mail: patrick.sequeira@zmk.unibe.ch

ABSTRACT

BACKGROUND: Systematic reviews are not an assembly of anecdotes but a distillation of current best available evidence on a particular topic and as such have an important role to play in evidence-based healthcare. A substantial proportion of these systematic reviews focus on interventions, and are able to provide clinicians with the opportunity to understand and translate the best available evidence on the effects of these healthcare interventions into clinical practice. The importance of systematic reviews in summarising and identifying the gaps in evidence which might inform new research initiatives is also widely acknowledged. Their potential impact on practice and research makes their methodological quality especially important as it may directly infuence their utility for clinicians, patients and policy makers. The objectives of this study were to identify systematic reviews of oral healthcare interventions published in the Journal of Applied Oral Science (JAOS) and to evaluate their methodological quality using the evaluation tool, AMSTAR.

METHODS: Potentially eligible systematic reviews in JAOS were identifed through an electronic search of the Scientific Electronic Library Online (SciELO). Details of the relevant aspects of methodology as reported in these systematic reviews were extracted from the full text publications. Methodological quality was assessed independently by two reviewers using the AMSTAR questionnaire.

RESULTS: Five systematic reviews were identifed, one of which was subsequently excluded as it was a review of a diagnostic test. Summary AMSTAR scores for the four included reviews were: 1, 5, 2 and 4 out of a maximum score of 11 (range 1-5, mean 3) with only one of the reviews scoring 5.

CONCLUSION: AMSTAR evaluation of the methodological quality of the relatively small number of systematic reviews published in JAOS illustrated that there was room for improvement. Pre-publication and editorial appraisal of future systematic reviews might beneft from the application of tools such as AMSTAR and is to be recommended.

Key words: AMSTAR. Systematic reviews. Quality. Methodology.

INTRODUCTION

Substantial progress appears to have been been made recently in Brazil in the feld of healthcare research. Haddad⁸ (2008) reported that this has placed it "among the 25 most productive countries in the world and Brazilian dentistry now ranks 5^th in recognized world scientific production". However whilst there has been a significant increase in the number of research publications concerns have expressed as to whether quality is keeping pace with quantity¹⁵. As published research increases there is an increasing requirement not only for periodic assessments of its quality but also for its integration and synthesis, so that it can be used as the basis for more well-informed healthcare policy making, and ultimately guide best practice in the provision of healthcare.

Systematic reviews have an important role to play in evidence-based healthcare. For example, reviews of healthcare interventions help clinicians to understand and translate the current best evidence of the effects of these interventions into their daily clinical practice. The importance of systematic reviews in summarising evidence and identifying gaps which might inform new research initiatives is widely acknowledged. This impact of systematic reviews on practice and research makes their methodological quality especially important, as it may infuence their utility for clinicians, patients and policy makers¹².

An assessment of the methodological quality of systematic reviews examining the effectiveness of oral and dental healthcare interventions reported that a substantial majority of these reviews were of mediocre quality⁷. A total of 65 reviews were included for assessment spanning publication years of 1990-2001. Major weaknesses highlighted by the investigators were inadequate search strategies; a lack of quality assessment of included studies, no examination of heterogeneity and inappropriate pooling of data.

Six years on from the Glenny⁷ (2003) article and a thought provoking editorial "Systematic reviews in endodontics - examples of GIGO?" echoed similar concerns and indicated that at least in the feld of Endodontics there had been limited improvement and that somewhat "alarmingly, the quality of these recent publications is negatively correlated to the frequency"²³. In an attempt to provide robust guidance the National Institute for Health and Clinical Excellence (NICE) in UK published a review examining the key components that comprise a methodologically sound systematic review. This study covered healthcare in general and included relevant work in oral healthcare¹⁹.

In 2002, the Agency for Healthcare Research and Quality (AHRQ) undertook a review of methods that could be used to rate the strength of scientific evidence, which included those used to evaluate the quality of systematic reviews. A number of such tools were identifed in which the key domains were: study questions, search strategy, inclusion and exclusion criteria, data abstraction, study quality and validity, data synthesis and analysis and funding or sponsorship of reviews²⁴. One of these tools, the Overview Quality Assessment Questionnaire (OQAQ) had been developed several years previously by Oxman and Guyatt¹⁴ (1991). Integration of this questionnaire with a quality checklist¹⁷ led to the development of a new instrument which could be used to assess the methodological quality of systematic reviews (AMSTAR)²¹.

AMSTAR is an 11-item assessment tool with good face and construct validity, has recently been validated and is being used increasingly by healthcare policy makers, health technology assessment agencies and some authors and journal editors²².

The main use of AMSTAR has been in the assessment of reviews of interventions rather than those covering other aspects of health or health care such as diagnosis, prognosis and etiology. It is used by agencies such as the Canadian Agency for Drugs and Technologies in Health³ in the evaluation of a range of systematic reviews but its use in the assessment of systematic reviews of oral healthcare interventions appears to have been sporadic and limited². However concerns have been expressed recently about the inappropriate use of AMSTAR in an assessment of the methodological quality of systematic reviews in Endodontology⁶.

Brazil has a prodigious number of dental schools, increasing from 32 in 1960 to 191 in 2009¹⁸. Many of these dental schools publish a dental journal and there are more than 90 such journals. Three of these; Brazilian Dental Journal; Journal of Applied Oral Science; Brazilian Oral Research are currently indexed to MEDLINE.

There has been an increased level of interest in evidence-based dentistry in Brazil, which has occurred alongside a steady rise in the number of systematic reviews published in some Brazilian dental journals; but little is known about the methodological quality of these reviews.

OBJECTIVES

To assess the methodological quality of systematic reviews of oral healthcare interventions published in the Journal of Applied Oral Science using the AMSTAR tool.

METHODS

Systematic reviews published in JAOS in recent years were identified through the electronic searching of the Scientific Electronic Library Online²⁰. This electronic virtual library is devoted to ensuring access to scientific information generated in low- and middle-income countries and provides access to full text articles of a substantial number of e-journals from Latin America, South Africa and elsewhere. The methodological quality of the included reviews was assessed using the AMSTAR quality assessment tool ("A Measurement Tool to Assess Systematic Reviews") (Figure 3).

Inclusion and exclusion criteria

Reports of systematic reviews were considered eligible for inclusion if they met the following criteria: the terms or phrases "systematic review", "meta-analysis" or "overview" were used in the title or abstract, or if the main text provided a clear indication that a systematic review had been carried out. In view of the wide and at times indiscriminate use of the terms "systematic review", only studies whose overall methods and conduct defined them as a systematic review were included whereas studies which contained for example only a literature search, systematic or otherwise, were excluded. The key components that comprise a methodologically robust systematic review were identifed, and then discussed and agreed on by the authors through consensus and were used as criteria for selection of reviews to be included in this study. These comprised; a focused research question, a comprehensive search, clearly defined study inclusion/exclusion criteria, transparent and reproducible selection of studies, a full assessment and reporting of the methodological quality of the included studies, independent data extraction, and plausible analysis and synthesis of data¹⁹. Reviews of a diagnostic test were excluded because the applicability of AMSTAR for reviews of observational (diagnostic, etiological and prognostic) studies has not yet been assessed²¹. If a systematic review did not involve human dental or oral tissues it was also excluded.

Electronic searches and assessment of eligibility

Two investigators (PS and ZF) independently examined the titles and abstracts of all 37 issues (volume 11; Jan/Mar 2003 to volume 17) of the JAOS available on-line and up to December 2009.

Full text copies of potentially eligible studies were obtained and assessed independently against the inclusion criteria by two investigators (ZF and VJ). Any disagreements on eligibility were discussed and if required were resolved by consulting with the third investigator (MS).

Assessment of methodological quality

AMSTAR was applied independently by two assessors (VJ and ZF) to each of the included systematic reviews and any differences in their evaluations were discussed and agreed through consensus. Individual items were categorized as "yes", "no"; "can't answer" or "not applicable" and the assessment data were tabulated. Although weights are not attached to any of the individual items in AMSTAR, summary quality scores were calculated, in keeping with other assessments that have used AMSTAR^2,21. Final grading of the methodological quality of each systematic review was based on the overall score and reported as either "high", "medium" or "low"; in concordance with the rating system used by the CADTH³.

RESULTS

The 37 issues of the JAOS provided a total of 604 titles and abstracts to publications, and most of which were original research reports. Five systematic reviews were identifed, one of which was subsequently excluded¹⁶ as it was a review of a diagnostic test. Four systematic reviews (Figure 1) were included and provided data for this study^1,4,5,11. The individual assessments for these reviews are presented in (Figure 2).

Out of the four reviews included in our study, and based on our AMSTAR assessments, three were categorized as of "low" quality based on their reports in JAOS. Although fairly comprehensive searches i.e. in terms of databases used, were conducted by three of the four reviews, the search strategies were only completetly described in one of the reports. Ideally, selection of studies and data extraction should be conducted by two independent reviewers but this was only adequately reported in two of the reviews. Although all of the reviews provided a list of included and excluded studies, only one review clearly indicated the reasons for exclusion of individual studies. The methods used to combine the results of studies differed quite markedly between the reviews. None of the reviews provided any reassuring evidence that reasonable attempts had been made to assess clinical diversity between the studies i.e. to assess whether the studies are similar enough to combine, and that any pooling of outcomes data across the included studies would be plausible. This evaluation should include an exploration of the characteristics of the eligible studies to assess the similarities and differences among the types of participants, the interventions received and the outcome measures being analysed.

Selection of eligible studies for a review should be based on clearly stated prespecifed criteria however the unavailability of protocols for any of these systematic reviews precluded any assessment of whether an a priori design had been used. The possibility of publication bias among eligible studies and its potential effects were not assessed in any of the reviews and there were no disclosures of conficts of either the authors of these reviews, and no indication of whether they had noted any potential conficts of interest by the investigators in the studies included in the reviews.

DISCUSSION

The methodological quality of the four systematic reviews published in the JAOS between 2003 and 2009 was rated as "low" for three of them and "medium" for Mickenautsch, et al.¹¹ (2007) using the AMSTAR assessment tool.

Only one of the four reviews identifed in our study attained a score of 5/11 and few of the reviews adequately satisfed key quality items such as the use of an a priori design; assessment of the quality of included studies and the assessment of publication bias. Yet these are arguably some of the most important prerequisites of a high quality systematic review.

In considering the results of this study it should be kept in mind that there is an important difference between the conduct of a systematic review and its reporting, and which may inadvertently result in a downgrading of any quality assessment. An example where a discrepancy may occur is when a journal in question limits the space available to publish the methods of research studies. In order to compensate some journals provide unlimited space, often online, to publish the methods of the research studies, including systematic reviews, that they contain.

Strengths

One of the strengths of this study is that we were able to identify and examine all systematic reviews in a consecutive sample of issues of JAOS from 2003 to 2009. This was possible because of online access to the relevant issues of the journal.

The simplicity of the design and application of the AMSTAR tool was important, because it made it quick and easy to use. The fact that it has been validated, gives credibility to its use as a tool to assess quality.

Limitations

One of the limitations of our study was the reliance on the summary score from AMSTAR to place articles into different quality categories. Although it is generally agreed that summary scores may mask inherent strengths and weaknesses of systematic reviews²² they can also prove helpful in informing decision making. For example, to help policy makers and clinicians CADTH categorizes AMSTAR quality assessments into three ranges, HIGH (9-11), MEDIUM (5-8), and LOW (0-4)³. Furthermore, the findings from an AMSTAR evaluation can also be used to explore the quality of a review within specific domains.

A further constraint is the very nature of a retrospective study as a point of application for the AMSTAR tool, or indeed any quality assessment. The assessment of the quality of a systematic review should, ideally, be carried out by assessing the methods actually used and not a report (prepublication or otherwise) of its conduct. In addition the publication of a protocol for a review prior to the full evaluation of the available studies has the advantage that the impact of review authors' biases and the potential for duplication are reduced, the transparency of methods and processes are promoted, and peer review of the planned methods can occur¹⁰.

Context

This study has highlighted that the methodological quality of the reviews published in the JAOS during 2003 to 2009 leaves the way open for improvements in either the reporting of systematic reviews, their conduct or both. This is important if the raised awareness of the concept of evidencebased dentistry within Brazil leads to a continuing increase in the number of systematic reviews being conducted and published by Brazilian researchers not only regionally but also internationally.

Within these reviews, an assessment of the quality of included studies for risk of bias can help to identify faws and errors in those studies, and the reporting of these assessments can help users of the review to decide for themselves about the weight to give to individual studies and the review as a whole. These biases within included studies can operate in a variety of ways on the results of a study, which can lead to over or underestimation of the true difference between the interventions being compared and potentially a distortion of the overall estimate for the review if data from those studies are synthesized in a meta-analysis. Inappropriate synthesis of data from studies with wide clinical diversity is seen in the literature¹³ and this was exemplifed in one of the reviews assessed in this study in which data from different interventions was pooled to provide a summary statistic. Before meta-analysis is considered a thorough appraisal of possible sources of clinical heterogeneity should be undertaken.

It was not the objective of our study to compare previous publications of methodological quality assessment of systematic reviews. A systematic review of this topic, including relevant oral healthcare publications has already been undertaken¹⁹.

A number of resources are available to help authors to conduct and report high quality well-designed, transparent and reproducible systematic reviews. One such resource, that is required by several journals, and provides a minimum set of items for reporting in systematic reviews and meta-analyses is PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)⁹. The conduct of a review is considered to be more important than its reporting quality, and therefore aids which help researchers to do high quality reviews should be welcomed.

Implications for research and publication policy

Tools, such as AMSTAR, exist to assess the quality of reports of systematic reviews, and their conduct if sufficient information is reported in the review. The use of robust methods in the review process will help to ensure that the findings are reliable and that the conclusions drawn can be applied with a degree of confidence. We believe that an AMSTAR assessment of systematic reviews could be used as part of the peer reviewing process for systematic reviews, with decisions on acceptance for publication being infuenced by whether or not the essential domains for quality have been adequately satisfed.

Busy clinicians may believe they do not have the time to assess the quality of published "systematic reviews". We disagree with this premise and affrm that they should make strident attempts to critically appraise all relevant research when making decisions about patient care. Healthcare providers need reliable information about the beneficial and harmful effects of healthcare interventions if they are to provide the highest quality care to their patients. Therefore it is crucial that published reviews are of high quality and are well-designed, ensuring that ultimately they provide a balanced and impartial summary of the results reported by taking into account any inconsistencies in the totality of the evidence. A quality assessment using AMSTAR may guide this critical appraisal.

We suggest that the use of a tool such as AMSTAR in the pre-publication and editorial appraisal of systematic reviews would help to ensure that those that are published are of high quality and are reported clearly, completely and in such a way that this quality is readily apparent.

CONCLUSION

An AMSTAR evaluation of the methodological quality of the four systematic reviews published in JAOS from 2003 to 2009 showed that there was room for improvement in their reports and, possibly, their conduct.

Disclaimer

Although some of the authors are members of The Cochrane Collaboration, funded by the National Institute of Health Research (NIHR) or employed by the Universities of Manchester or Bern, the views expressed in this paper are not necessarily the views or the official policy of The Cochrane Collaboration, The NIHR, The University of Manchester or the University of Bern.

ACKNOWLEDGEMENT

We are grateful to Mike Clarke, Professor of Research Methodology at Queen's University Belfast, Northern Ireland for comments on an earlier version of this article.

Received: March 01, 2011

Modification: July 14, 2011

Accepted: July 22, 1011

Publication Dates

History