Background
The updating guidelines working group
Objectives
-
Primary objectives:
-
○Estimate average time for recommendations to become out of date.
-
○Evaluate two alternative search strategies to assess the validity of CPGs recommendations.
-
-
Secondary objectives:
-
○Design a strategy to assess the validity of CPGs recommendations.
-
○Evaluate resources used to perform each strategy.
-
○Assess the agreement between study participants in identifying references that potentially could update CPGs recommendations.
-
Methods
Design
Population and eligibility criteria
Strategies
Stages | What | How | Who |
---|---|---|---|
Stage 1 | Identification of clinical questions and recommendations | Review original CPG | Guideline methodologist from research group |
Stage 2 | Baseline survey | Recommendation basal survey (http://www.surveymonkey.com) | Clinical experts |
Stage 3 | Update exhaustive literature search | - Recover original exhaustive literature search | - Guideline methodologist from original CPG |
- Define the filters that will be used: 1) study design; 2) publication date | - Information specialist | ||
Stage 4 | References database by clinical questions | Reference management software | Information specialist |
Stage 5 | First references screening | Reference management software | Guideline methodologist from research group |
- Topic | |||
- Study design | |||
- Publication type | |||
Stage 6 | References matching | Reference management software | Guideline methodologist from research group |
Stage 7 | Recommendations database | Statistic program | Guideline methodologist from research group |
Stage 8 | Second references screening |
Stage 8a Reference survey to assess the updating effect: feasibility test | - Guideline methodologist from original CPG - Guideline methodologist from research group |
- Identification of relevant references | |||
- Identification of key references | |||
Stage 8b Sample size calculation | Statistician | ||
- Assess the potential changes in the recommendation | |||
Stage 8c Reference survey to assess the updating effect (pdf forms) | - Guideline methodologist from original CPG | ||
- Guideline methodologist from research group | |||
- Clinical experts | |||
Stage 9 | Final report | Final report with study results | Guideline methodologist from research group |
- Recommendations still valid | |||
- Recommendation needed update |
Stages | What | How | Who |
---|---|---|---|
Stage 1 | Identification contents | Review original CPG | Guideline methodologist from research group |
Stage 2 | PLUS search | - Choose existing MeSH and SNOMED in PLUS database related with original CPG contents | PLUS information specialist |
- Define the filters that will be used: 1) population; 2) study purpose categories; 3) publication date | |||
Stage 3 | Reference databases by CPGs | Reference management software | PLUS information specialist |
Stages | What | How | Who |
---|---|---|---|
Stage 1 | Identification clinical questions | Review original CPG | Guideline methodologist from research group |
Stage 2 | Clinical questions eligibility | Include clinical questions with ≥ two explicitly PICO components | Guideline methodologist from research group |
Stage 3 | ReSe |
Stage 3a Development ReSe serches | Guideline methodologist from research group |
Stage 3b Evaluation ReSe serches | |||
Stage 3c Refinement ReSe serches | |||
Stage 3d Define the filters that will be used: 1) study design; 2) publication date | |||
Stage 4 | References database by clinical questions | Reference management software | Guideline methodologist from research group |
Strategy to assess the validity of recommendations
-
Stage 1: Identification of clinical questions and recommendations. We will extract the clinical questions, the recommendations (identified in the “Summary of recommendations” section) and their strength (SIGN [16] or GRADE [17] system) for each original CPG. Recommendations will be numbered and classified (prevention, screening, diagnosis or treatment).
-
Stage 2: Baseline survey. Using a similar approach as Shekelle et al. [10] we will conduct a survey by e-mail (http://www.surveymonkey.com) with clinical experts for each CPG. They will evaluate whether they consider that recommendations are up to date and if they know any new studies that might change the recommendations (Additional file 2).
-
We will perform the survey in a convenience sample of six clinical experts who participated in the CPG development. Original guideline methodologists will identify the survey participants: 1) four clinical experts representing the different areas covered by the guideline; and 2) two external clinical experts.
-
Stage 3: Update literature search. We will recover the original exhaustive literature searches per clinical questions.
-
Information specialists, preferably from the original guideline, will run the searches in the databases and apply the corresponding study design filters (randomised controlled trials [RCTs] or SRs) used in the original searches. Date filters will be established from the complete year in which the original search was completed onwards.
-
Step 4: References database by clinical question. We will cluster the references obtained from the baseline survey and from the search. We will identify and eliminate duplicates and build a database by clinical questions with the references identified.
-
Step 5: First reference screening. We will evaluate whether references are pertinent to the topic of interest, the study design (RCTs or SRs) and the publication type (we will include original articles or abstracts from conferences about original studies) (Additional file 3).
-
Step 6: Reference matching. We will match pertinent references with one or more related recommendations.
-
Step 7: Recommendations database. We will analyse the references databases to obtain recommendations: 1) without references; 2) with low turnover (≤ median number of references per recommendation); or 3) with high turnover (> median number of references per recommendation).
-
Step 8: Second reference screening. We will design a recommendation form to sort out the pertinent references identified (Additional file 4). The form will contain: 1) relative to each recommendation: clinical question, recommendation, evidence quality and strength of recommendation; 2) relative to the related references : citation, ± PubMed Unique Identifier (PMID), abstract and study design; and 3) relative to the assessed references: a question to identify relevant references (references that could be use when considering the update of a recommendation but not necessarily trigger a potential update), a question to identify key references (references that could potentially trigger a recommendation update) and a question to assess the potential changes in the recommendation (in relation with population, intervention, comparison, outcome, quality of evidence, direction and/or strength of the recommendation [18]).
-
We will send the recommendations forms to clinical experts and guidelines methodologist by e-mail (we will schedule three remainders every two weeks). Each form will be assessed by two clinical experts and one guideline methodologist. The disagreements will be resolved by consensus.
-
Step 9: Final report. We will prepare a final report with recommendations that may need updating, in relation to the new evidence identified. The final report will be sent the corresponding institutions that developed these guidelines and the clinicians who will collaborate in the study.
PLUS search strategy
-
Stage 1: Identification topics. We will extract the topics for each original CPG (identified in “table of contents”).
-
Stage 2: PLUS search. PLUS information specialists will develop the corresponding search strategies by matching existing Medical Subject Headings (MeSH) and Systematized Nomenclature of Medicine (SNOMED) with the CPGs topics. They will perform the searches applying PLUS population, study purpose categories (therapy/prevention; diagnosis; prognosis; etiology; economics; clinical predication guide; differential diagnosis) and publication date filters. No filter will be applied to select for either original or review articles.
-
Stage 3: References database by CPGs. PLUS information specialists will obtain a database of references by CPG.
Restrictive search strategy
-
Stage 1: Identification of clinical questions. We will extract the clinical questions for each CPG.
-
Stage 2: Clinical questions eligibility. Restrictive searches will be structured taking into account the PICO (population, intervention, comparator and outcome) structure of each clinical question. To develop each strategy we will include at least two PICO components from each question and their corresponding most representative keywords. The questions that do not explicitly include PICO components will be excluded.
-
E.g. an explicit clinical question from the CPG for Prostate Cancer Treatment is “In patients with prostate specific antigen (PSA) relapse after radical prostatectomy, what kind of salvage intervention is safer and more effective?”. A non explicit clinical question would be “What is the safest treatment and most effective option for a patient with prostate cancer at the locally advanced clinical stage?”. In this question treatment alternatives are not clearly defined and make it a very broad question to be answered by the ReSe strategy [19].
-
Stage 3: ReSe. To develop ReSe, based on original exhaustive search strategy, we will: 1) Select MeSH terms: If available, for each keyword we will find the most specific MeSH term (e.g. "Prostate-Specific Antigen" MeSH term for the population of the question “In patients with PSA relapse after radical prostatectomy, what kind of salvage intervention is safer and more effective?” [19]); 2) Select free text words [Tw]: for each keyword we will select the most relevant specifics free text words and search them in title (e.g. we would select “prostate[ti] AND specific[ti] AND antigen[ti]” free text words for the question “In patients with PSA relapse after radical prostatectomy, what kind of salvage intervention is safer and more effective?” [19]).
-
We will evaluate if the ReSe retrieves all original references considered in the recommendations of the original CPGs. We will evaluate this by calculation the proportion of original references which are retrieved (sensitivity). If a ReSe search does not find all the original references (sensitivity <100%) we will refine it until it retrieves them all.
-
For the refinement, if needed, we will be using one or both of the following options: 1) use of less specific MeSH terms; and/or 2) free text words to search in title or abstract. We will limit each ReSe by type of design. For each ReSe we will apply the filter Therapy of the Clinical Study Categories of Clinical Queries, using both narrow and broad scope, and we will apply the SR filter developed at the Health Information Research Unit, McMaster University [20]. Finally, we will perform the searches applying publication date filters.
-
Step 4: References databases by clinical questions. For each clinical question we will obtain three databases, one using the therapy filter plus narrow scope, one using the therapy filter plus broad scope, and one using the SR filter.
Outcomes
Primary outcomes
-
Average time for recommendations to become out of date.
-
Proportion references that trigger a potential recommendation update (key references) identified by the alternative search strategies.
Secondary outcomes
-
Resources used by strategy (time and participants).
-
Agreement between clinical experts and guideline methodologists across references screening.