Compliance of systematic reviews in ophthalmology with the PRISMA statement

Systematic reviews and meta-analyses are key study designs that are used to summarize published research findings that are replacing traditional reviews and expert commentaries. In these studies, the authors collect and critically assess all evidence that fits pre-specified criteria with the aim of answering a specific research question. They are therefore able to limit biases by including a wider range of populations than can be included in a single centre study. Moreover, they may quantify the heterogeneity in the available data and allow the results of multiple studies to be pooled into a meta-analysis, thereby increasing statistical power. In many cases, the goal is to identify the most appropriate treatment for a particular clinical paradigm [1‐3]. Reliance on systematic reviews and meta-analysis by health care practitioners and policy makers, including those involved in ophthalmology research, in which approximately 1000 systematic review publications have been published, is increasing [4].

The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement was published in 2009 and consists of a 27-item checklist that covers each section of an article and includes a flow diagram. Its purpose is to help authors improve the quality of reporting in systematic reviews and meta-analyses [5]. Similar studies have been reported in different fields, including orthorhinolaryngology [6], orthodontics [7], radiology [8], gastroenterology and hepatology [9], and orthopaedics [10]. In a previous study, we also assessed the compliance of articles related to plastic surgery to the PRISMA statement. Our results showed that the reporting quality in plastic surgery was suboptimal [11].

Regardless of the field of research, poorly reported systematic reviews can be misleading and potentially dangerous in clinical practice. Studies in ophthalmology may present additional challenges to both researchers and readers because of the need for complexity in study designs. For example, there are two potential data points in each patient (i.e., two eyes), and accounting for this fact may require alternative designs and methods of analysis. Hence, the results and conclusions of such studies may be confusing if they do not provide an appropriate level of explanation [12]. Systematic reviews play a role in enabling the accurate appraisal of the literature because they allow large pools of data to be simultaneously interpreted, which can resolve some of the complexity in this field. It is also important for the authors of these studies to provide complete, clear and transparent information by using good reporting methods [5, 13]. The PRISMA statement is an objective index that can be used to determine whether systematic reviews are adequately performing this role.

The aim of this study was to determine the reporting quality of systematic reviews and meta-analyses in ophthalmology according to the PRISMA statement. We evaluated articles published between 2010 and 2015 from five major journals with the highest impact factor.

This study is registered at Research Registry (unique identifying number: researchregistry952; http://​www.​researchregistry​.​com). Institutional review board approval was not required because the study did not involve human participants.

Among the 5 journals included in this study, only one journal (Ophthalmology) required the PRISMA checklist to be used to report a systematic review or meta-analysis.

The initial electronic search identified 253 articles. Of these, 115 were selected for data extraction (Fig. 1). Arbitration was required to determine whether to include 8 articles, and disagreement occurred regarding 375 points across the 115 articles. The Cohen’s Kappa statistic was 0.65, which indicates a good level of agreement between scorers.

The median compliance across the 115 included articles was 15 out of 27 items (56%) and had a range of 5–26 (19–96%) and an inter-quartile range of 10 (37%). The compliance of the articles with each PRISMA item showed wide variance (Table 1). More than half of the articles achieved more than 50% compliance with the PRISMA checklist. However, approximately 10% of the articles failed to achieve compliance higher than 30%, and no single article met all 27 criteria (Fig. 2).

Meanwhile, compliance with individual items was variable, with three items achieving a level of compliance equal to or greater than 90%, while compliance was less than or equal to 30% for 5 items. The compliance was highest for items related to ‘the description of the rationale for the study’ (item 3, 100%). This was followed by the ‘general interpretation of results’ (item 26, 96%) and the ‘inclusion of a structured summary in the abstract’ (item 2, 90%). Compliance was poorest for ‘indication of review protocol and registration’ (item 5, 9%), ‘specification of risk of bias that may have affected the cumulative evidence’ (item 15, 24%), and ‘description of clear objectives in the introduction’ (item 4, 26%).

The median compliance was 15 out of 27 (56%), which is relatively low in comparison with most previous studies that investigated the reporting quality of systematic reviews in different specialties [8‐11, 18]. Half of the articles (62 articles) achieved a compliance of less than 60% (Fig. 2), and no single article met all 27 criteria of the PRISMA statement. Considering that only one journal required the use of a PRISMA checklist prior to the submission of a manuscript presenting a systematic review or meta-analysis, there is a significant amount of room in which improvement can be made in reporting the quality of systematic reviews in ophthalmology.

Five items with compliance less than or equal to 30% include items related to describing the review protocol and registration, describing the risk of bias, describing the objectives using PICOS, and describing additional analysis (items 5, 15, 4, 23, and 22, respectively). Interestingly, poor compliance with these items has also been noted in other previous studies that examined the reporting quality of systematic reviews in different specialties [6‐11, 18]. Regardless of their overall compliance with the PRISMA checklist, previous studies in specialties including orthorhinolaryngology [6], orthodontics [7], and radiology [8] all achieved compliance lower than 30% in items relating to describing the review protocol and registration and the risk of bias. A study of journals in gastroenterology and hepatology [9] showed that there was generally high median compliance with the PRISMA checklist (above 80%) but poor compliance (less than 20%) in describing the protocol and registration. Similar studies that evaluated orthopaedic journals [7], our previous study that assessed the compliance of plastic surgery articles to PRISMA [11], and a study of nursing journals [18] produced overlapping results that showed there was low compliance regarding items related to the protocol and registration, the description of bias, and the description of additional analyses. Moreover, a review of 300 systematic reviews [19] found that just under a quarter (23.1%) of authors reported assessing possible publication bias and that no single article in the study mentioned the review protocol or review registration.

The generally low compliance observed in this study is an indication that there is a lack of awareness regarding the importance of review protocols and registration information among authors. The use of protocol and review registration is important because a protocol allows more accurate comparisons to be made between authors and leaders and supports better transparency during the review process [5, 20]. There are web database registration sites that authors can easily use to obtain information regarding protocol and registration. These include the PROSPERO [20] and Research registry [21] databases, which we recommend. It is also disappointing that most specialties achieve a low level of compliance in items related to the assessment of risk of bias. The attempt to eliminate bias is a very fundamental component of randomized controlled trials, as is the assessment of bias in systematic reviews and meta-analysis [22]. Therefore, the poor compliance with items relating to assessing bias in most specialties is actually surprising. We suspect that this is caused by a lack of awareness regarding essential points of research reporting among authors because we noticed that many of the articles briefly mentioned ‘bias’ but then failed to describe the methods or results of their assessment of the risk of bias.

While improved awareness among stakeholders (i.e., journal reviewers and editors, funders, institutions, and readers) is required, we reiterate our previous call for journals to make the PRISMA checklist mandatory for the electronic submission of systematic reviews [11]. Enforcing these guidelines at the time of journal submission may improve the quality of reporting, and their endorsement has been shown to improve adherence [9]. Unfortunately, it is difficult to comment on this point with regard for the five ophthalmology journals included in this study because only one of them required the PRISMA checklist. However, it was recently demonstrated that journals increased their endorsement of reporting guidelines in their introduction to the authors between 2011 and 2014 (10% vs. 19%) [23]. The number of systematic reviews and meta-analyses published in the field of ophthalmology has dramatically increased over the last decade [24, 25]. While this is a positive indication that there is adequate recognition of the increasing number of original high-quality articles published in this field, it also means that there is increased room for error due to the large amount of data. Increased importance should therefore be placed on accurate and detailed reporting [25]. Clear and transparent information not only allows an adequate interpretation of results but also helps authors to correct errors during earlier stages of the study [5, 13].

Some limits need to be discussed regarding the methods used in our research. Because we focused on journals with high impact factors, our findings do not necessarily reflect adherence to PRISMA for systematic reviews published in journals with low impact factors. Additionally, the articles were electronically searched instead of handsearched, the latter of which would have ensured that we retrieved all articles that met the eligibility criteria. However, it was difficult to obtain access to paper copies of the journals, and using electronic search and screening procedures reduced the time that was consumed during the data search phase of this study. Additionally, we believe that MEDLINE and EMBASE cover a sufficient breath of articles (each contains over 20 million records) [26, 27].

We assigned equal weights to all items in the PRISMA checklist and used the same number of denominators in each article. However, this imposed a limitation on our study in that certain items on the checklist may have more impact than others on the reporting quality of systematic reviews. For example, it may be useful to weigh items related to the description of detailed methods more heavily than those related to the identification of the report as a systematic review in the title. Additionally, certain elements in the PRISMA checklist did not apply in some articles. For example, if the systematic review did not include a meta-analysis, items relating to meta-analyses, such as the presentation of the results of each meta-analysis (item 21), are not applicable to the review. Although we considered the article as fulfilled if it justified why meta-analysis was not performed, most articles failed to state such justification. Furthermore, the scoring of some items could vary according to the publishing style of each journal. For example, item 2 requires authors to provide a structured summary that includes many characteristics, such as the background, objective, data sources, and study eligibility criteria. However, compliance with this item can vary depending on how many words are used and what structure each journal requires the authors to provide. We attempted to be generous with such elements, but it remains possible that some articles may have lost a mark as a result of this type of limitation.

Systematic reviews in ophthalmology involve methodological issues that are specific to eyes and vision search, e.g. unit of analysis; we have not explicitly evaluated reporting of such items in this study. Because PRISMA was designed to cover a wide range of specialties, it has some limitations with regard for covering the specific details required in each specialty. In ophthalmology, PRISMA does not consider presentation of two sets of data, one from each eye (i.e., the right and left eye of each patient), a point that becomes even more complicated in studies wherein data can be obtained for only one eye in some patients [2].

Potential barriers to compliance with the PRISMA checklist may include a lack of awareness or understanding of the PRISMA guidelines, which we suspect is the most common cause of lack-of-compliance [28]. However, further study is required to assess such barriers among key stakeholders in ophthalmology because no research into these barriers to compliance with PRISMA has been published.

The reporting quality of systematic reviews and meta-analysis in ophthalmology needs to be significantly improved, especially in describing the review protocol and registration, describing the risk of bias, describing the objectives, and describing additional analysis. We strongly suggest that journals require that all PRISMA criteria should be fulfilled before allowing electronic journal submission because we believe that this will eliminate ambiguity in the reporting of studies. Additional research aimed at identifying potential barriers to compliance may be required to improve compliance with the PRISMA guidelines.