Background
Methods
Search strategy
Inclusion and exclusion criteria
Population | Included: adult symptomatic patients (with symptoms being indicative of cancer) presenting at primary care or patients referred with symptoms indicative of cancer |
Excluded: asymptomatic patients (screening population). | |
Technology | Included: Diagnostic prediction models, based on 2 or more featuresa, that estimate the risk of prevalent but undiagnosed colorectal cancer. |
Excluded: prognostic or screening prediction models Statistical tools that estimate the probability of developing cancer over a defined period of time. Prediction models that did not include colorectal cancer. | |
Setting | Included: primary care |
Excluded: secondary care; on-line tools developed for use by the general population | |
Study design | Included: - any design for the development, validation or accuracy of diagnostic prediction models (as defined under ‘Technology’); - comparative studies of diagnostic tools that assessed impact in clinical practice (Randomised controlled trials, controlled before-after, and interrupted time-series; studies analysing national trends in cancer diagnosis before and after diagnostic tools became available) |
Excluded: uncontrolled studies reporting qualitative data | |
Comparison | Usual care or the use of another diagnostic tool |
Outcomes | For studies reporting development, validation and/or accuracy of prediction models: Estimates of the risk of being diagnosed with cancer (e.g. ORs, HRs) AND/OR Any details on the development, validation or accuracy of the tool: • Model development: method; assumptions; predictors; shrinkage; coefficient weighting • Model evaluation (validation) • Assessing (quantifying) model performance: discrimination (ability to discriminate participants with or without the outcome, e.g. area under the ROC curve); calibration (agreement between predicted and observed outcome); overall performance (for discrimination and calibration, e.g. R2); classification (e.g. sensitivity, specificity, predictive values) For studies reporting evaluations of the impact of tools: Primary outcomes - patient-related outcome measures (including the number of cancer diagnoses, time to cancer diagnosis, stage of cancer at diagnosis, resection rates, patient health-related quality of life, other patient-reported outcome measures); - survival; - economic outcome measures (resource use, cost per diagnosis), cost per QALY; Secondary outcomes - referral patterns. |
Exclude: models that report the risk of survival (or stage at diagnosis etc.) | |
Publication type | Included: Published in full and in English |
Excluded: commentaries, letters |
Selection of studies
Data extraction
Risk of bias assessment
Synthesis
Results
Studies identified
Development/validation studies
Prediction models
Prediction model | Number and category of descriptors | Stage of development | Study design | Country | Population | Source |
---|---|---|---|---|---|---|
Colorectal cancer | ||||||
Bristol-Birmingham equation | 8 Symptoms, Test results | External validation | Retrospective Case-control | UK | Derivation cohort: THIN Validation cohort: CAPER | Marshall 2011 [29] |
External validation | Prospective cohort | The Netherlands | CEDAR study: Patients referred to endoscopy centres by participating Dutch primary care practices. 2009–2012 | Elias 2017 [27] | ||
Netherlands model | 3 Symptoms, Patient demographics | Apparent performance | Prospective cohort | The Netherlands | 290 consecutive patients with rectal bleeding presenting to 83 GPs in Limburg (Netherlands) September 1988 to April 1990Predictors: Questionnaires completed by GPs and patients, and laboratory test results. | Fijten 1995 [28] |
External validation | Prospective cohort | UK | patients referred from primary care with colorectal symptoms over a 3-yr period to the Leighton Hospital, Crewe, Cheshire, UK | Hodder 2005 [40] | ||
External validation | Prospective cohort | Netherlands | CEDAR study: Patients referred to endoscopy centres by participating Dutch primary care practices. 2009–2012 | Elias 2017 [27] | ||
Machine learning algorithm | Numerous models are reported Patient demographics, Symptoms, Medical history, Test results | Apparent performance | Case-control | The Netherlands | anonymised electronic records from two GP database systems from the Utrecht region, Netherlands, between 01 and 07-2006 and 31-12-2011 | |
Danish model | 2 Patient demographics Symptoms | Apparent performance | Prospective cohort | Denmark | Patients presenting to GPs with first episode of rectal bleeding. Study 1: 750 GPs 1989–1991 Study 2: 450 GPs 1991–1992 | Nørrelund 1996 [31] |
External validation | Prospective cohort | The Netherlands | CEDAR study: Patients referred to endoscopy centres by participating Dutch primary care practices. 2009–2012 | Elias 2017 [27] | ||
Qcancer | 6 (females) 7 (males) Symptoms, Medical history, Test results | Internal validation | open Prospective cohort | UK | QResearch database | Hippisley-Cox 2012c [22] |
External validation | Prospective cohort | UK | THIN database | Collins 2012 [23] | ||
RAT (2005) | 10 Symptoms, Test results | Apparent performance | Case-control | UK | Patients attending all 21 general practices in Exeter, Devon, UKCases identified from the cancer registry at the Royal Devon and Exeter Hospital | Hamilton 2005 [33] |
External validation | Prospective cohort | The Netherlands | CEDAR study: Patients referred to endoscopy centres by participating Dutch primary care practices. 2009–2012 | Elias 2017 [27] | ||
RAT (2009) | 8 Symptoms, Test results | Apparent performance | Case-control | UK | THIN database | Hamilton 2009 [43] |
RAT (bowel) | 10 Symptoms, Test results | Apparent performance | Case-control | UK | GPRD (currently called the CPRD) | Stapley 2017 [35] |
Metastatic cancer | ||||||
RAT | 7 Symptoms, Test results | Apparent performance | Case-control | UK | Patients attending 11 general practices in Devon, UK | Hamilton 2015 [36] |
Multiple cancer sites | ||||||
Qcancer (female) | 7 (uterine) 10 (breast, blood) 11 (ovarian, renal) 12 (cervical) 13 (colorectal, gastro-oesophageal) 14 (pancreatic) 15 (lung) 22 (other cancers) Medical history, Symptoms, Test results, Patient demographics | Internal validation | Open prospective cohort | UK | QResearch database | Hippisley-Cox 2013 [38] |
QCancer (male) | 3 (testicular) 8 (renal tract) 12 (colorectal) 13 (gastro-oesophageal) 14 (prostate, blood) 15 (pancreatic) 17 (lung) 20 (other cancers) Medical history, Symptoms, Test results, Patient demographics | Internal validation | Open prospective cohort | UK | QResearch database | Hippisley-Cox 2013b [37] |
Muris abdominal complaints model | 5 Symptoms Patient demographics Test results | Apparent performance | Prospective cohort | The Netherlands | Patients presenting to GPs for new abdominal complaints. 1989 | Muris 1995 [30] |
(Netherlands) | External validation | Prospective cohort | The Netherlands | CEDAR study: Patients referred to endoscopy centres by participating Dutch primary care practices. 2009–2012 | Elias 2017 [27] | |
Abdominal model, Holtedahl and colleagues (2018) | 4 Symptoms, Patient demographics | Apparent performance | Prospective cohort | Norway, Denmark, Sweden, Scotland, Belgium, Netherlands | GP records from the participating countries | Holtedahl, 2018 [39] |
Critical appraisal
Model (author offirst version) | Stage of development covered | I. Participantselectiona | II. Predictorsa | III. Outcomea | IV. Samplesize andparticipantflowa | V. Analysisa |
---|---|---|---|---|---|---|
Apparent performance | ✓ | ? | ✓ | ? | x | |
External validation (colorectal only) [27] | ✓ | ✓ | ✓ | ? | ? | |
Internal validation | ✓ | ✓ | ✓ | ✓ | ✓ | |
External validation (colorectal only) [23] | ✓ | ✓ | ✓ | ✓ | ✓ | |
Bristol-Birmingham (Marshall) [29] model for colorectal cancer | ||||||
External validation | ✓ | ? | ✓ | ? | ✓ | |
External validation (Elias and colleagues,2017) [27] | ✓ | ✓ | ✓ | ? | ? | |
Netherlands’ (Fitjen 1995 [28]) model for colorectal cancer | ||||||
Apparent performance | x | ✓ | ✓ | ? | x | |
External validation (Hodder and colleagues,2005) [40] | x | ? | x | ✓ | ? | |
External validation (Elias and colleagues,2017) [27] | ✓ | ✓ | ✓ | ? | ? | |
Netherlands’ (Kop) [32] ‘machine learning’ for colorectal cancer | ||||||
Apparent performance | ✓ | ? | ✓ | ? | ? | |
Danish (Nørrelund 1996 [31]) model for colorectal cancer | ||||||
Apparent performance | ✓ | ? | ✓ | ? | x | |
External validation (Elias and colleagues,2017) [27] | ✓ | ✓ | ✓ | ? | ? | |
Netherlands’ (Muris 1995 [30]) model for abdominal complaints | ||||||
Apparent performance | ? | ✓ | ✓ | ? | x | |
External validation (Elias and colleagues,2017) [27] | ✓ | ✓ | ✓ | ? | ? | |
Prediction model for abdominal cancers(Holtedahl and colleagues, 2018) [39] | ||||||
Holtedahl, 2018 | Apparent performance | ? | ✓ | ? | x | ? |
Performance of the models
Prediction model | Validation (using derivation or external dataset) | Dataset used, country | AUC (95% CI) | Source |
---|---|---|---|---|
Colorectal cancer | ||||
Bristol-Birmingham equation [29] | Derivation | THIN, UK | 0.83 (0.82, 0.84) | [29] |
External | CAPER, UK | 0.92 (0.91, 0.94) | [29] | |
External | CEDAR, Netherlands | 0.84 (0.77, 0.90) | [27] | |
Netherlands model [28] | Derivation | Primary care, Netherlands | 0.97 | [28] |
External | Secondary care, UK | 0.78 (0.74, 0.81) | [40] | |
External | CEDAR, Netherlands | 0.72 (0.62, 0.81) | [27] | |
Netherlands model including polyps [28] | Derivation | Primary care, Netherlands | 0.92 | [28] |
Qcancer (male) [22] | Derivation | Qresearch, UK | 0.91 (0. 09, 0.91) | [22] |
External | THIN (multiple imputation), UK | 0.92 (0.91, 0.92) | [23] | |
THIN (complete case analysis), UK | 0.90 (0.89, 0.91) | [23] | ||
Qcancer (female) [22] | Derivation | Qresearch, UK | 0.89 (0.88, 0.90) | [22] |
External | THIN (complete case analysis), UK | 0.91 (0.90, 0.92) | [23] | |
Danish model [31] | External | CEDAR, Netherlands | 0.6 (0.48, 0.72) | [27] |
RAT (2005) [33] | External | CEDAR, Netherlands | 0.81 (0.75, 0.88) | [27] |
Multiple cancer sites | ||||
Muris abdominal complaints model [30] | External | CEDAR, Netherlands | 0.62 (0.54, 0.70) | [27] |
Impact studies
Study ID | Prediction tool | Country of tool development | Tool description |
---|---|---|---|
Hamilton and colleagues, 2013 [55] | RAT presented on a mouse mat and desk top flip chart (for lung and colorectal cancer) | UK | The RAT algorithm is displayed in a table/matrix format, which allows a risk estimate to be calculated for a single symptom, pairs of symptoms or repeat attendances with the same symptom. The values are colour-coded to aid interpretation. |
Emery and colleagues, 2017 [56] | Education resource card containing the RAT and referral guidelines | UK (RAT), Australia (guidelines) | Resource card containing the RAT tables for colorectal, lung and prostate cancer, as well as the Australian National Breast and Ovarian Cancer Centre guidelines for investigating new breast symptoms |
Price and colleagues 2019 [16] | RAT and/or QCancer in any form (e.g. paper, software etc.) for any cancer | UK | Any affirmative GP practice access to RAT and/or QCancer |
Critical appraisal
Randomsequencegeneration | Allocationconcealment | Baselineoutcomemeasurementssimilar | Baselinecharacteristicssimilar | Incompleteoutcomedata | Knowledge ofthe allocatedinterventionsadequatelypreventedduring the study | Protectionagainstcontamination | Selectiveoutcomereporting | Other risksof bias | |
---|---|---|---|---|---|---|---|---|---|
Randomised controlled trials | |||||||||
Emery 2017 [56] | ✓ | x | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Pre-post study | |||||||||
Hamilton2013 [55] | N/A | N/A | N/A | N/A | ? | N/A | N/A | ? | x |
Cross-sectional survey | |||||||||
Price andcolleagues2019 [16] | N/A | N/A | N/A | ? | ? | ✓ | N/A | ✓ | x |
Study outcomes
Study ID | Prediction tool | Country | Study design | Intended purpose | Main results for colorectal RAT |
---|---|---|---|---|---|
Hamilton 2013 [55] | RAT for lung, colorectal cancer in two formats: mouse mat and desk top flip chart | UK | Pre-post study | To compare referrals and investigations for colorectal and lung cancer before and after the implementation of RATs | 26% increase in 2-week referrals (1173 before, 1477 after); 15% increase in colonoscopies (1762 before, 2032 after) No conclusion possible on the effectiveness of the intervention |
Emery 2017 [56] | Education resource card including RAT for colorectal, lung and prostate cancer | Australia | Factorial cluster RCT | to measure the effect of community-based symptom awareness and GP-based educational interventions on the time to diagnosis (i.e. TDI) for patients presenting with breast, prostate, colorectal or lung cancer in rural Western Australia | No significant differences in the median or ln mean TDI at either intervention level: Colorectal cancer: -GP intervention vs control: median TDI 124 vs 122 days; ln mean difference − 0.03 95% CI − 0.51–0.45 P = 0.42 -community intervention vs control: median TDI 107 vs 133 days; ln mean difference − 0.26 95% CI − 0.63–0.11 P = 0.16; |
Price 2019 [16] | Access to any RAT and/or Qcancer tool in any format | UK | Cross-sectional survey at GP practice level | To compare the mean 2WW referral rates between GP practices reporting access to RAT and/or Qcancer and those who reported no access to these tools | No statistically significant difference between mean referral rates between practices reporting access or no access to RAT and/or Qcancer: mean difference of 3.1 referrals per 100,000 population (95% CI − 5.5, 11.7, p-value 0.48) |