Critical appraisal of clinical practice guidelines for the management of COVID-19: protocol for a systematic review

The present study will use the AGREE II instrument for the quality assessment of CPGs focused on TI use and/or the management of adult patients with COVID-19 [10]. The present protocol was prospectively registered on the international Prospective Register of Systematic Reviews (PROSPERO) with the following registration number: CRD42020219944. The present protocol has been reported in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) [17] (see PRISMA-P checklist in Additional file 1). The completed systematic review will be reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [18]. If significant deviations are made from this protocol, then they will be reported and published with the results of the review.

The criteria for eligible COVID-19 CPGs were developed in accordance with the Population, Intervention, Comparison and Outcomes (PICO) framework [19]. A CPG is a systematically developed guidance document that makes recommendations for a specific condition based on an analysis of benefits and risks of each intervention [20]. Alternatively, when the evidence for interventions are limited such that an assessment of benefits and harms cannot be accurately made, consensus statements may be developed during a convention of experts [20]. Recommendations are made in a consensus statement if there is sufficient agreement between the expert panel regarding the interventions; however, these recommendations may be biased by those involved in the recommendation development process [20]. Despite these differences between CPGs and consensus statements, both are types of guidance documents [21]. We will include CPGs that focus on adults (18 years or older) with COVID-19. With respect to the interventions, we will include CPGs that primarily discuss and make recommendations for TI use and/or the management of COVID-19. In terms of the TIs that are eligible, we will include all TIs used for the treatment and/or management of COVID-19. These include, but are not limited to, tocilizumab, recombinant angiotensin-converting enzyme 2, remdesivir, lopinavir/ritonavir, anticoagulants, antiviral combination therapies, hydroxychloroquine/chloroquine compounds, colchicine, corticosteroids, oseltamivir/amantadine, and convalescent plasma. We will not limit the inclusion of CPGs to specific TIs, as doing so would prevent a comprehensive assessment of CPGs focused on the management and/or TI use for COVID-19. We will include CPGs that are evidence-based, publicly available, published after 2019, and in English. We will exclude articles that focus on the diagnosis or screening of COVID-19 and the associated SARS-2-CoV-2 virus. COVID-19 diagnosis is commonly done through viral tests such as antigen testing immunoassays and nucleic acid amplification tests [22]. Antigen immunoassays are often employed as a screening tool as they are easy to produce, detect the presence of specific antigen, and indicate current infection [23]. In contrast, nucleic acid amplification tests are viral diagnostic tests that detect the presence of viral genetic material by identifying the ribonucleic acid comprising its genome [23, 24]. CPGs that are not in English or have a more recent update available from each organization will also be excluded. Furthermore, we will also exclude articles that are abstracts, editorials, letters, position papers, protocols, consensus statements, conference proceedings, and summaries of CPGs.

The primary outcome of this study is the assessed quality of eligible CPGs. The assessed quality will be determined through the calculation of the scaled domain percentages for each CPG, as outlined in the AGREE II instrument [10]. We will also report the average appraiser scores and average overall assessments, in addition to the scaled domain percentages. The secondary outcomes of this study are the types of management and/or TI recommendations made. We will determine which recommendations are inconsistent, unique, or duplicate across CPGs.

Two reviewers will independently perform title and abstract screening using Rayyan [26]. Articles that meet the eligibility criteria will be included and their respective full-texts will be retrieved for full-text screening. Full-text screening will be similarly performed independently and in duplicate. Any discrepancies regarding the eligibility of an article will be resolved through discussion and consensus between reviewers, or recruitment of another co-author for resolution. Supplementary materials and related documents for CPGs that pass full-text screening will be retrieved thereafter. A PRISMA flow chart illustrating the details of the selection process will be provided [18].

The reviewers will independently and in duplicate perform data collection and extraction in data extraction spreadsheets prepared a priori. Any discrepancies in the data extracted will be resolved through discussion and consensus between reviewers, or recruitment of another co-author for resolution. For eligible CPGs, we will data extract demographic information including the author, title, year of publication, publishing organization, country, and funding source. Any management and/or TI recommendations made will also be extracted in addition to their corresponding level of evidence and grade of recommendation.

We will assess eligible CPGs using the AGREE II instrument which is a well-established and widely-validated international tool used to assess the quality and reporting of CPGs [10]. It is composed of 23 items over six quality domains which include the following: (1) scope and purpose, (2) stakeholder involvement, (3) rigor of development, (4) clarity and presentation, (5) applicability, and (6) editorial independence [10]. The instrument uses a seven-point Likert scale that ranges from strongly disagree (1) to strongly agree (7) that each item is met [10]. Each eligible CPG will be evaluated by four appraisers as recommended by the AGREE II instrument [10]. Prior to assessing the CPGs, each reviewer will complete the AGREE II Tutorial + Practice Exercise on the ‘My AGREE PLUS’ platform [27]. The reviewers will also perform a minimum of three rounds of calibration with CPGs that vary in quality. The reviewers will then score the CPGs for each of the 23 items over 6 domains. The overall quality of the CPG (1 to 7) will sequentially be evaluated by the reviewers and used to recommend for or against the use of a CPG. Any major discrepancies in the scoring of a CPG (greater than 3 points) will be resolved through discussion and consensus between reviewers, or recruitment of another co-author for resolution. We will then calculate the average overall assessment scores and average appraisal scores for each CPG. The average overall assessment scores will be calculated by averaging the scores each appraiser gave for ‘overall guideline assessment.’ The average appraisal scores will be calculated by averaging the score an appraiser gave for all 23 items, and then subsequently averaging this value calculated for each of the 4 appraisers. The quality of each domain will be compared within and across CPGs using the scaled domain percentages. The scaled domain percentages will be calculated through the summation of the appraisers’ scores for the items within each domain, and subsequently scaling this value as a percentage of the maximum possible score for the given domain.

Descriptive statistics will be used to quantify the quality of the CPGs. Specifically, we will calculate the average overall assessment scores, average appraisal scores, and the standard deviation for each of these two scores. The scaled domain percentages will also be calculated. We will tabulate these results for ease of quality comparison both across and within the domains of the CPGs. A narrative synthesis will be provided to explain the characteristics and findings of the included CPGs.

We will tabulate the recommendations made in each CPG for the management of COVID-19. Additionally, we will compare inconsistent and duplicate TI and/or management recommendations made between CPGs. Specifically, we will indicate whether a therapy was recommended for or against use in a table. If a therapy was recommended, then it will be indicated with a green box. If a therapy is recommended against, then it will be indicated with a red box. If a therapy recommendation is unclear or conflicting, then it will be indicated with a yellow box. This will allow for ease of comparison between therapy recommendations across the eligible CPGs. In addition to comparing the CPGs in a table, we will also provide a narrative synthesis to describe and compare the recommendations of the included CPGs.

There are several notable strengths to this study, including the use of a comprehensive systematic review to determine eligible CPGs pertaining to TI use and the management of COVID-19, use of multiple appraisers, and the use of the well-established and widely-validated AGREE II instrument.

There also exist some limitations inherent in our proposed study. Firstly, CPGs that are not in English should be excluded; however, this may result in the omission of CPGs pertinent to the proposed research question. Secondly, the target population of the CPGs for this study is all adults over 18 years of age; however, we do not intend on separately analyzing quality between CPGs for adults (18–65 years) from CPGs for elderly adults (65 years and above). It is possible that CPG quality may differ between these target populations. Lastly, given the COVID-19 pandemic, CPG developers may seek to rapidly disseminate knowledge to better manage COVID-19. As a result, CPG developers may prioritize the rapid dissemination of new CPGs instead of prioritizing the methodological quality of CPGs.

Our proposed systematic review and assessment will aide healthcare professionals who wish to improve the standard of care for patients with COVID-19 by incorporating CPGs into their practice, guideline developers in the creation of CPGs or improvement of existing ones, researchers who want to identify gaps in knowledge, and policy-makers looking to encourage and endorse the adoption of CPGs into clinical practice. Additionally, the evidence gathered through this systematic review will provide insight into gaps in available CPGs which may result in the identification of other research avenues.

Any amendments made to the proposed study will be indicated on PROSPERO and in the “Methods” section of the completed systematic review. This systematic review will be disseminated in a peer-reviewed journal and may be presented at local, national, or international conferences.