An appraisal of analytical tools used in predicting clinical outcomes following radiation therapy treatment of men with prostate cancer: a systematic review

The search strategy resulted in 165 potentially relevant abstracts/articles and these were reduced to 72 once duplicates were removed and title and abstracts were screened (Fig. 1). The full-text of these papers was reviewed against the inclusion criteria (reasons for exclusion are reported in Additional file 1: Table S1a and b) and 47 finally selected. Study recruitment periods varied considerably with the earliest patients being from 1984 [10] and the latest 2009 [10‐13] (Table 1). The populations of individual studies varied from 80 [14] to 7,839 [14, 15] with a combined population of 60,457 (Tables 2, 3 and 4). The majority of studies were retrospective (n = 38), however seven studies recruited prospective cohorts (for one study [16] it was not stated whether it was retrospective or prospective).

The 47 papers finally included in this review described 97 individual predictive models. Of these models, 16 related to brachytherapy treatment (Table 2), 72 to external beam radiation therapy (Table 3) and nine to a combination of brachytherapy and external beam radiation therapy (Table 4).

Across all radiation treatment modalities, outcomes relating to PSA levels post treatment were most common (39 models) followed by prostate cancer specific mortality (29 models). Measures of metastases (17) and overall survival (14 models) were less common (note that some papers report more than one outcome and model). Of those studies reporting development of new models (66), only nine reported validation either internally or in an additional cohort. Only 67/97 (69%) models included variables which were considered to be readily available in existing data sets.

Critical appraisal considered the criteria set by the CEBM appraisal tool for prognostic studies [9]. Risk of bias ranged from moderate (Q1; Was the defined representative sample of patients assembled at a common point in the course of their disease? (72%), Q2; Was patient follow-up sufficiently long and complete? (64%)) to low (Q3; Were outcome criteria either objective or applied in a ‘blind’ fashion? (85%), Q4; If subgroups with different prognoses are identified, did adjustment for important prognostic factors take place? (91%)) (Table 5).

Since the publication of previous reviews, there has been considerable progress in the field of outcomes prediction following prostate cancer treatment. This review identified 47 papers published between 2007 and 2015, which describe 97 predictive tools for men receiving radiotherapy. This includes 66 models which were newly developed and 31 which were validations of already published predictive tools. Consistent with previous reports, most tools (65%) are yet to be validated in a population outside the derivation set. Studies were included from 2007 as the modality of radiation therapy has changed significantly over the past decade, and historic data may not be a useful basis for prognosis. Apart from modality, the total dose has also significantly increased however, we found that only five studies [13, 16, 20‐22] did not use data from men treated as far back as the 1990s.

The volume of research carried out in the field of prognostics has exploded over the last decade. A systematic review that included all studies published before July 2007 (the cut-off date for inclusion in the present review) identified 17 studies on prognostic models that related to prostate cancer patients treated with radiotherapy [4]. In this review 39 new studies were identified which investigated prognostic markers for BCF. Unfortunately, the majority of new studies did not undertake validation, mirroring the finding of the previous systematic review. As validation – particularly external validation – is vital for the appropriate clinical implementation of prognostic models, this suggests that resources and efforts are not being efficiently targeted to improve tools available for clinical practice.

With regards to the methodological quality of the literature, our critical appraisal found that overall studies were at low to moderate risk of bias. The greatest risk was created by insufficient follow-up (defined as a mean or median of ≥5 years) which only occurred in 64% of studies. There was also a moderate risk of bias created by the possibility of included patients being at different points in the course of their prostate cancer, however in the majority of cases this was due to insufficient specificity in the description of inclusion criteria as opposed to reported differences. There was little risk of bias created by the measurement of outcomes, as the main outcomes (biochemical failure [various definitions], metastasis, survival) were objective, or by a lack of adjustment for important prognostic factors as the essential factors of prostate cancer prognosis (PSA, Gleason score, and clinical stage) were used nearly universally.

Model accuracy was not reported in 57% of the models included. Model accuracy was reported to be highest in Vainshtein 2014 [23] with a c-index of 0.86 derived for prediction of prostate cancer specific mortality with the CAPRA score (originally established in [24]), including the addition of variables for the presence of Gleason 5 and treatment with ADT (this c-index relates to patients not receiving ADT). This study acts to externally validate the CAPRA scoring system (with modifications) in patients treated with external beam radiation therapy, though this improvement to the score requires further validation in other populations. Of the remaining 42 models which reported predictive accuracy, c indices were typically in the 0.70–0.80 range which would be considered ‘reasonable’ according to Hosmer and Lemeshow [25]. Notably, those papers which did not report external validation typically had higher c-indices suggesting that original model developments should be considered optimistic in their predictive capacity. The lowest c-index (0.49, 95%CI 0.37 to 0.61) was reported for a study [26] performing external validation of the Prostogram nomogram (originally established in [27]) suggesting this nomogram may have little predictive value.

The predictive tools identified in this review included joint-modelling approaches but not neural networks which have featured in previous reviews. This may reflect a change in statistical tools available since publication of earlier catalogues [4]. Two of the survival models [28, 29] did not account for competing risks when predicting prostate cancer specific mortality, a potential weakness which could easily be addressed.

The majority of papers attempted prediction relating to biochemical recurrence, prostate cancer specific mortality or overall survival with a smaller subset predicting metastases. Sixteen of the 97 models identified related to brachytherapy with 72 for external beam radiation therapy and 9 a combination of the two. This could reflect more wide-spread use of external beam radiation therapy, and we might anticipate more tools relating to HDR brachytherapy (with or without EBRT) in the future. There is a dearth of externally validated nomograms focusing on brachytherapy and brachytherapy in combination with external beam radiation therapy particularly looking at overall survival and cancer specific survival outcomes.

This study did not explicitly set out to uncover tools incorporating novel variables, but only those which could be used in current clinical settings. Despite this, 31% of studies included reference to variables which have been less studied to date (e.g. mid-point PSA levels). While such variables may prove useful, there is currently limited opportunity to validate these observations using existing datasets. It is possible that additional variables including standardised measures of comorbidity, imaging features or genetic markers, which are becoming more accessible may help to improve the accuracy of future models. For a recent review of potential molecular and genetic candidate see Hall et al. 2016 [30].

Most predictive tools identified in this review were developed in US populations. This observation should be considered by clinicians who are based outside the US when selecting a predictive model to assist treatment decision making. Where possible, tools validated in a setting similar to one’s own clinical practice should be selected for use. The number of tools available internationally would be increased with additional validation work conducted outside the US and particularly in multi-national cohorts.

We observed a large degree of variation in the quality of reporting clinical predictive tools. This may stem from the fact that authors are not aware of reporting guidelines in the field or indeed that such guidelines exist. The TRIPOD guidelines (http://​www.​equator-network.​org/​reporting-guidelines/​tripod-statement/​) for reporting of multivariable prediction models were published in March 2015, shortly before the cut-off for papers included in this review. These guidelines have been widely endorsed and published in key journals [31‐39]. Further publication of multivariable models would benefit greatly from adherence to these guidelines.

Tools which aid decision making offer more accurate prediction of clinical outcomes when compared to clinical judgement alone. This understanding has led to a large increase in the number of predictive tools relating to clinical outcomes post radiation therapy between 2007 and 2015. This review identifies 47 papers describing 97 models published in the period, a substantial increase compared to the 17 models previously described between 1966 and 2007. Of the models identified, 65% had no external validation and 57% did not report accuracy. Thirty one percent of models included variables which are not part of typical registry data sets, and are therefore difficult to validate. Despite these limitations, there are accurate and externally validated models for external beam radiation therapy treatment which predict prostate cancer specific mortality. There are fewer models which accurately predict outcomes following brachytherapy (alone or in combination with external beam radiation therapy). This review provides an accessible catalogue of predictive tools which could be used currently (i.e. those with high accuracy after external validation) and identifies those which should be prioritised for future validation.