Risk prediction models for colorectal cancer in people with symptoms: a systematic review

We used a combination of subject headings including ‘colorectal cancer’, ‘risk/risk factor/risk assessment/chance’ and ‘prediction/model/score’ to conduct an electronic literature search within Medline and EMBASE. The search period ran from January 2000 to March 2014 (see Additional file 2 for the complete search strategy for Medline and EMBASE). We subsequently hand searched the reference lists of all included papers. We also considered for inclusion papers published before 2000 describing the development of models that were validated in included papers.

To be included, studies had to be published as a primary research paper in a peer-reviewed journal and either describe, validate or assess the impact of a risk model that allowed identification of people at higher risk of CRC or CRC and advanced colorectal neoplasia. The risk model had to feature two or more risk factors, including symptoms, for prevalent undiagnosed colorectal cancer at the level of the individual. In addition, a quantitative measure of model performance was required. Conference proceedings, papers not in English, and studies of a specific patient group, for example immunosuppressed patients or patients with a past history of CRC, were excluded.

One reviewer (JUS) screened the titles and abstracts of papers identified by the Medline and EMBASE searches to exclude studies that were clearly not relevant. A second reviewer (TGSW) independently assessed a random selection of 10 % of the papers at title and abstract level and both reviewers (TGSW and JUS) independently assessed all the full text of papers if a definite decision to reject could not be made based on title and abstract alone. All reviewers met to discuss discrepancies and reach a consensus decision on inclusion or exclusion.

Two reviewers (TGSW and JC) extracted data from each paper using a standardised form. Discrepancies were examined and resolved by a third reviewer (JUS). We extracted information on the components of each risk model and potential sources of bias. These included: study design and participants; methods of model development; and variables included in the risk model. The methods of studies published for each risk model was also classified according to the TRIPOD guidelines (1a-Development only; 1b-Development and validation using resampling; 2a-Random split-sample development and validation; 2b-Non-random split-sample development and validation; 3-Development and validation using separate data; 4-Validation study) [17]. Where multiple models were described within the same study, each model was included separately.

Reported measures of discrimination (area under the receiver operating characteristic curve (AUROC)), accuracy (sensitivity and specificity), calibration and utility were used to compare the performance of different risk models and thresholds in development and validation populations. Numerical values for the AUROC were used to compare discrimination and the sensitivity and specificity to compare the accuracy, of different models and thresholds. For those papers in which sensitivity and specificity were not reported explicitly, where possible we calculated the values from data provided in the paper. Figures were produced using RevMan version 5.3 and where multiple studies reported the sensitivity and specificity of the same model at the same threshold, the combined values were calculated using Meta-DiSc version 1.4.

Quality assessment was performed at the same time as data extraction. Since our review included studies with different designs we used a checklist based on the Critical Appraisal Skills Programme guidelines for case-control and cohort studies [18] as an initial framework, and then classified each study as high, medium or low quality. No studies were excluded based on quality assessment alone.

After duplicates were removed, the search identified 12,808 papers of which 12,765 were excluded at title and abstract level. A further 29 were excluded after full-text assessment by at least two reviewers (TGSW and JUS). There was complete agreement among researchers throughout the screening process. The most common reasons for exclusion were that the papers did not report a statistical measure of model performance (n = 9), only evaluated one predictor (n = 6), or were conference abstracts (n = 4). Three additional papers were identified through citation searching, including one published prior to 2000 which was included as it had been externally validated in one of the papers identified through the literature search. In total we included 18 papers describing 15 risk prediction models in the review (Fig. 1). Only one paper assessed the impact of one of the models in practice [19].

A summary of the quality assessment of each of the 18 papers is given in Table 1. Eleven were assessed as high quality, four as medium quality and three as low quality. The studies assessed as low quality were two cross-sectional studies recruiting patients presenting to either primary or secondary care with rectal bleeding [20, 21], and the single study assessing model impact [19].

Table 1 also summarizes the methods used to develop and validate the models. Of the 15 included models, nine were developed in primary care populations [21‐27] and six in secondary care [20, 28‐32]. Most (n = 11) had CRC as a single outcome, whilst two in secondary care predicted CRC combined with advanced adenoma (a polyp measuring 10 mm or bigger, or a polyp of any size with a villous histology [20]) or pre-malignant adenomas [28]. The remaining two reported CRC risk alongside the risk of cancers of several other sites [26, 27].

Most models were developed from either cross sectional (n = 6), prospective cohort (n = 4) or case-control studies (n = 4), with one developed based on clinical experience [30]. Four used self-administered questionnaires or interviews conducted by non-medically trained staff to gather information. The remaining 11 models required the input of a healthcare worker. Nine of the 15 models have been validated: one using bootstrap resampling [32]; two using a random split-sample [26, 27]; and six in external populations [21, 24, 25, 30]. Details of the methods and study populations for the validation studies are also given in Table 1.

We categorized risk factor variables into five types: demographic, personal and family medical history, symptoms, signs, and investigations (Table 2). Seventeen variables were included in three or more models: four demographic variables (age, sex, smoking, alcohol); family history of CRC; eight symptoms (rectal bleeding, change in bowel habit, diarrhoea, constipation, abdominal pain, weight loss, loss of appetite, mucous in the stool); abnormal rectal examination; and three investigations (haemoglobin, mean cell volume, faecal occult blood testing). All models included symptoms, four included only symptoms [22‐24], and most also included age (n = 11) and sex (n = 9). The choice of variable in each model was often influenced by the study design. For example, the Hamilton et al. 2005 risk model [22] and the CAPER score [24] were developed from case-control studies using primary care records and include symptoms plus additional signs and investigations without any demographic information or personal or family medical history. In contrast, the models developed by Adelstein [31, 32] from patient-completed questionnaires include symptoms plus demographic information and personal or family medical history.

The main strengths of this review are the broad search strategy and the systematic approaches used to identify studies and extract data. However, as with all systematic reviews, our conclusions are limited by the quality of published research. The included studies were heterogeneous in design, setting and duration of follow-up. Additionally, although we only included risk models with published performance data, two only provided positive predictive values, only six have been validated in external populations, and only one has been assessed for clinical utility or impact. This tendency for research into risk prediction tools to focus on model development rather than validation and impact is well documented [39, 40]. Nevertheless, it limits the conclusions that can be made about the potential role of these risk models in clinical practice.

Where data on discrimination and accuracy was reported there is little to distinguish between the models developed and validated in primary care. The four QCancer® models [25‐27] were developed and validated using prospective cohort designs within two large UK primary care record databases, QResearch (development) and THIN (validation). All contain a combination of demographic, symptom, and investigation variables routinely recorded in electronic medical records, and all have AUROCs above 0.89 in either split-sample or external validation, and sensitivities around 0.7 with specificities over 0.95. The BB equation was also developed in the THIN database but using a case-control design [24] and has been validated in a dataset of paper-based primary care records from 21 English primary care practices, whilst the CAPER score was developed from those same primary care records and validated in the THIN database [24]. Both the BB equation and CAPER score performed better in the primary care record dataset, achieving comparable AUROCs and sensitivity and specificity to the QCancer® models. As Marshall et al. [24] describe, reasons for this may include the fact that clinical features of colorectal cancer were identified from both paper and electronic records and included analysis of free text in the primary care record dataset. The use of case-control designs instead of cohort studies for both the development and validation of these risk models, however, means that these measures may not accurately reflect their performance in population based cohorts due to the wide dispersion of risk factors in the cases and controls and the restricted distribution of matched variables.

Additionally none of these models have been validated outside data routinely collected by General practitioners (GPs). It is known that some symptoms are more likely to be recorded by a GP in patients in whom cancer is suspected. For example, patients coded as having a change in bowel habit are at greater risk than those with diarrhoea or constipation [41]. As a result of this coding bias it is likely that the recorded symptoms used in these models overestimate the significance of those symptoms in the presenting population. Whilst all these models in primary care can therefore accurately discriminate between patients in whom GPs have or have not chosen to routinely record these symptoms and could be used to identify those in whom further investigation or referral is necessary, how they perform in the consultation setting when GPs are having to decide whether the patient in front of them does or does not have a given symptom is not known.

Models developed and validated in secondary care settings were instead all based on cross-sectional studies of patients referred with symptoms of CRC with data collected using patient and/or physician questionnaires at the time of investigation. All have similar discrimination (AUROC 0.8 to 0.9) in development populations, but the only model to be externally validated is the WNS developed by Selvachandran et al. [30]. This has been validated in four separate populations with a prevalence of CRC of around 5 %. The AUROCs range from 0.76 [35] to 0.86 [30] and sensitivity and specificity from 0.96 and 0.40 to 0.64 and 0.82. A low threshold, with a high sensitivity, could therefore be used to identify those in whom further investigation is not required. However, all the patients included in these studies had already been assessed as high risk by primary care physicians so the score would be likely to perform less well in an un-referred primary care population, therefore validation in that setting is required.

The risk models identified in this review have the potential to improve the diagnosis of CRC by helping clinicians to identify those patients presenting with symptoms of possible CRC in whom further investigation and referral is most appropriate. The potential advantages of risk prediction models in this context are that they can include combinations of symptoms and other risk factors, and different thresholds for action can be used. For example, a threshold with high sensitivity and high specificity could be used to define a high risk patients that require urgent referral, whilst one with very high sensitivity and low specificity could be used to identify those who do not require further investigation at that time.

Sackett and Haynes [42] identified four questions which must be addressed before incorporating diagnostic tests into clinical practice, however. The first three are concerned with test performance: whether test results are different between those with and without the condition; whether patients with certain test results are more likely to have the target disorder; and whether the test results distinguish patients with and without the target disorder. This review shows that risk models for CRC do exist which meet these criteria, with the best performing having sensitivities above 0.7, specificities above 0.9 and AUROCs over 0.9 in external validation studies.

Most contain variables that are easily obtainable in a single consultation and so could relatively easily be incorporated into practice. Whether any of them are any better than a clinician’s assessment is, however, uncertain. In the only study to compare a risk model with clinical judgement [30] the WNS was compared to the specialist clinical assessment of a comprehensive questionnaire-gathered history and there was no significant difference in discrimination. There is more evidence to suggest that the models are better than previous referral guidelines. Although not the primary aim of this review, in all cases where models were compared with guidelines the predictive models showed better discrimination and equal or better accuracy [11, 24, 30, 34, 35].

This review also cannot answer Sackett and Hayne’s fourth question - whether patients undergoing the diagnostic test fare better than similar untested patients. No studies have sought to address that. Before incorporating any of these risk models into practice, further research is therefore needed to validate the most promising models in clinical settings in comparison to clinical judgement and current referral guidelines, and to assess the impact of the use of these risk models in practice. Further work is also needed to consider whether CRC alone or in combination with advanced colorectal neoplasia or adenoma is the most appropriate outcome. This review focused on risk prediction models for CRC and only two models, which both reported only limited performance data and have not been validated, included advanced colorectal neoplasia or adenoma in addition to CRC [20, 28]. It is, therefore, not possible from this review alone to know how the performance of models predicting the combined outcome of advanced colorectal neoplasia and CRC compares to those with CRC as a single outcome. One study, however, reported the performance of risk models for CRC, advanced adenoma, or adenomas 6–9 mm in diameter separately within the same population [31]. The discrimination for the CRC model was substantially better (AUROC 0.87 compared to 0.70 and 0.67 for advanced adenoma and adenomas 6–9 mm in diameter respectively). This probably reflects the fact that many adenomas are asymptomatic and so identified less well by risk prediction models developed in symptomatic populations. The discriminatory performance for advanced adenoma in this symptomatic population is comparable with risk models developed for asymptomatic individuals [43]. This suggests that models with a combined outcome of advanced colorectal neoplasia and CRC may identify those with CRC less well than models with CRC as a single outcome. However, it is widely accepted that CRC arises from the adenoma-carcinoma sequence and so identification of patients with advanced colorectal neoplasia has the potential to reduce future incidence of invasive CRC. The choice of outcome(s) therefore depends on the purpose for which the risk models are to be used. If the priority is identification of patients with prevalent CRC, then a risk model including CRC as the sole outcome is likely to have the greatest discrimination and accuracy and allow targeting of referrals and further investigations most effectively. If the priority is instead to identify both prevalent CRC and those patients at high risk of developing CRC in the future, then a risk model including advanced colorectal neoplasia would be more appropriate.

The introduction of a two-step process into the recently updated NICE referral guidelines [44], in which the referral decision for individuals at intermediate risk is made based on the result of testing for occult blood in faeces, also provides an opportunity for research into incorporating other pre-referral tests into risk models. These include faecal immunochemical tests [16] and potentially more specialised tests, such as exfoliated DNA and carcinoembryonic antigen (CEA), which were of predictive value in secondary care developed models.

From work in other disease areas [45‐49] we know that uncertainty about how to account for risk factors perceived to be important but not included in the tools, and the perception that clinical judgement is as good as or better than risk tools, contribute to the low uptake of risk models. Practical issues such as lack of time, poor knowledge or understanding of the tools, and poor computer software also restrict model use. Additionally, a recent study using simulated consultations with risk prediction tools for cancer has shown that clinicians may interpret symptoms inconsistently, leading to inaccurate and unreliable cancer risk assessment, and GPs were reluctant to use the tools for fear of alarming their patients if the risk information is presented too explicitly [50]. Research is therefore also needed to understand how best to incorporate risk prediction models into routine practice, including communication of risk information to patients, and to address the barriers to their use.