Maintaining relevance in HIV systematic reviews: an evaluation of Cochrane reviews

The HIV field has high research outputs, and the cycle of questions, trials, and guideline modification is rapid. This raises the question of how to keep HIV systematic reviews up to date, relevant, and derived from the highest methodological standards [1, 2]. One approach is to regularly update reviews to incorporate new findings; and indeed, some questions endure and should be updated; and for others however, the review question may have been answered, changed substantially, or become irrelevant, and blindly updating such reviews can result in wasted effort and resources. To provide insight into how review questions change over time, we formally appraised the relevance, methods, and need to update a suite of HIV systematic reviews and protocols published in the Cochrane Library between 2000 and 2016.

Between 1 May 2016 and 30 May 2018, we critically appraised all published Cochrane HIV reviews and protocols based on the Cochrane Updating Classification System [3, 4]. The Updating Classification System acts as a guide to readers to assess whether a review is up to date, needs updating, or might need updating in the future [3, 4]. We modified the tool for assessing protocols. Reviews and protocols were appraised by evaluating: (a) the currency and priority of the review question (through discussion with experts in the field and exploring the literature), (b) the methods, by using an abbreviated version of the MECIR standards [2], and (c) whether there were new studies to be included in the review, or whether another high-quality review had already been published on the topic outside of Cochrane. Based on the outcome of this assessment, we determined whether reviews should be updated, and whether protocols should continue to full reviews. We categorized them as follows: (1) continue with review update or allow protocol to progress to full review and (2) not for update or protocol progression due to (a) methodological concerns, (b) outdated research question, or (c) review already up to date. At least two people read and appraised each review independently, and the classification was agreed in team meetings chaired by experienced systematic review authors. The tool is available in the Additional file 1.

We identified and appraised 109 Cochrane HIV reviews and 39 HIV protocols (148 articles in total), published on the Cochrane Database of Systematic Reviews between 2000 and 2016. Review and protocol topics included questions about HIV treatment interventions (63%), HIV prevention interventions (32%), health system interventions (2%), and interventions to improve HIV diagnosis (3%). The majority of reviews (N = 98; 66%) were published between 2010 and 2016, with a smaller proportion published between 2000 and 2009 (N = 50; 34%). We concluded that 19 (13%) of all reviews and protocols were eligible for an update or progression to full review, and the remaining 129 (87%) were assessed as “not for updating or progression”.

Of the 109 published systematic reviews, 95 were assessed as “not for updating” (87%). Of these, 62% had outdated research questions (Fig. 1) where the intervention was superseded by a superior intervention or by another more up-to-date review, or where questions were already answered, and there was little to no ongoing research in the topic area. For example, a review of the “Effectiveness of first-line tenofovir, emtricitabine and efavirenz for patients with HIV” would not require updating as this question has been answered. Another example is the use of HSV-2 suppressive therapy for HIV prevention: this is no longer considered an option for HIV prevention; although there may be some additional studies for inclusion, updating such a review would be of little value to consumers or policy makers. We discontinued 11 reviews due to methodological errors (37%): this was the result of inadequately conceptualized research questions (the population, interventions, comparison, or outcomes were not sufficiently characterized or relevant). Sometimes, this led to “empty” reviews (with no included studies after searches were conducted). Another methodological problem was review questions which were not unique to HIV populations, for example, reviews evaluating drug effects in HIV subpopulations with no clear rationale for why these agents would have different outcomes in HIV-positive people.

Thirty-four of 39 protocols were stopped from progressing to full reviews (87%) (Fig. 1). The two main reasons for discontinuation also included outdated research questions. This included protocols where the topic was no longer relevant to policy makers or consumers, or where an important HIV research area was evaluated but the protocol review question did not reflect current practices or thinking. The remaining protocols were discontinued due to methodological limitations, as a result of inadequately conceptualized research questions. Specifically, protocols encompassed a very broad set of treatment strategies that were difficult to combine in one review, or authors did not consider how to meaningfully combine complex HIV interventions. This was common in protocols which included behavioral HIV prevention interventions and pragmatic implementation trials. Thirty protocols were published more than 2 years previously (interquartile range 3–9 years).

This appraisal of Cochrane systematic reviews and protocols demonstrates that topics in the HIV field become outdated quickly, and this was by far the commonest reason for discontinuation of both reviews and protocols, including those from more recent years. In addition, changes in reporting standards over time resulted in outdated methodology. Protocols which progressed slowly became irrelevant before completion, and inadequately conceptualized research questions resulted in disorganized or empty reviews which, at times, did not address issues specifically relevant to HIV populations, or sufficiently characterize interventions and key HIV outcomes.

Evaluating the relevance of systematic reviews prior to updating is essential in all fields but particularly important where there is rapid development of new innovations and care strategies, as seen in HIV research. Where there are delays, protocols should be re-evaluated and adapted to ensure that final reviews answer current questions. With the recent drive to automate and create “living” systematic reviews with frequent updates [5, 6], it is essential that methods of maintaining currency are incorporated into these evidence synthesis approaches as questions may change or become irrelevant [7, 8].

We recommend that authors planning to conduct HIV systematic reviews reflect on how their review can contribute, by thoroughly exploring the current evidence, determining if primary studies exist on the topic of interest, identifying other systematic reviews already conducted, and clearly defining their research question and what their review may add. In addition to presenting the PICO (population, intervention, comparison, and outcomes) elements, authors should fully characterize the interventions of interest, limit the number and types of interventions, and determine how to synthesize results from studies with behavioral or complex interventions. Several available tools can be drawn on and adapted to aid the design of conceptual frameworks and characterization of interventions and complexity: this can help structure review questions to produce meaningful results [7‐11]. Review team effort and Cochrane editorial procedures need to ensure timely completion of reviews to avoid them being outdated before completion.

In summary, HIV research questions and the systematic reviews summarizing the relevant research evidence become outdated quickly, and rapid conduct and publication is critical; clearly defined research questions need to be regularly re-evaluated alongside changes in the field to maintain relevance, particularly in the era of living systematic reviews.