Accuracy of clinical predictions of prognosis at the end-of-life: evidence from routinely collected data in urgent care records

The study used anonymised data routinely collected as part of Coordinate My Care, the largest Electronic Palliative Care Coordination System in UK, serving London 2010 -2022 [22]. Coordinate My Care is designed to facilitate patients and their clinicians in the making of advance care decisions and to share them digitally in real-time with relevant healthcare professionals involved in patients’ care. Coordinate My Care includes clinical information; patients’ preferences about end-of-life care such as their “preferred place of care”, “preferred place of death”; ceiling of treatment and resuscitation decisions. Since 2017, Coordinate My Care plans have contained a mandatory field that requires a temporal clinical prediction of survival. Clinicians are asked to provide a “likely prognosis” using a predetermined list of outcomes including the following categories: “days”, “weeks, “months”, “year”, “years” and “undefined”.

Patients consent to the use of their anonymised information for research at the time of consenting to the creation of a Coordinate My Care record. This service evaluation was approved by the Royal Marsden Committee for Clinical Research (1002, approved 14/09/2020). This study was co-produced with input from patients and members of the public and supported through a Biomedical Research Centre at the Royal Marsden and Institute for Cancer Research grant for Patient and Public Involvement.

We included all individuals aged 18 or older with a Coordinate My Care plan registered between January 2010 and December 2020. The Coordinate My Care platform is connected to the NHS Spine, a digital NHS information exchange platform via which recorded dates of death are updated. Anonymised data were extracted in April 2021.

Stata (Version 14) was used for analyses. Demographics and clinical details were summarised for the study population and stratified according to whether the patient was alive or dead at the time of analysis. The overall cohort was also stratified into those with and without (missing) a defined survival estimate. Records containing a prognosis estimate were included in the analysis of accuracy (“Full prognosis cohort”). Records without a defined prognosis estimate were examined for differences compared to those with a documented prognosis.

Because the temporal estimates that clinicians used to make their prognostic estimates lacked clear boundaries (e.g., “weeks” is not precisely defined on the Coordinate My Care record) it was not possible to analyse the accuracy of predictions without making some assumptions about the boundaries of these categories. We defined a prognosis of “days” to mean 0–13 days, on the basis that if a patient survived for 14 days or more then they had already survived for “weeks”. Using similar logic, we defined “weeks” to mean ≥ 14 days and < 60 days (i.e., fewer than two months); “months” to mean ≥ 60 days and < 365 days (i.e., less than one year); and “years” to mean more than one year. As end-of-life care is typically focussed on patients in their last year of life, the fields “Year” and “Years” were combined to create a cohort of patients who had a longer prognosis. As well as having some face-validity, these boundary limits are like those used to define “days”; “weeks”, “months” and “years” in the well-validated Prognosis in Palliative care Study (PiPS) predictor models [9, 23, 24].

When more than one prognosis was recorded, the initial prognostic estimate was used for analysis. Survival was calculated as the number of days between the date of the initial survival prediction and the date of death if recorded. Survival predictions (“days”, “weeks”, “months”, “year/years”) were compared to individuals’ actual survival for the total study population and for groups of patients hypothesised to have distinct end-of-life illness trajectories [20]: cancer, dementia and frailty, heart disease and respiratory disease.

The accuracy of clinical prognostic estimates was defined as the frequency with which the clinician selected the correct survival category and was described as a percentage of the total number of estimates [9]. The extent of agreement between estimated and actual prognosis was quantified using linear weighted Kappa statistic. Kappa denotes the agreement between prognostic estimate and survival and considers that some prognoses could be expected to be correct purely based on chance. The kappa statistic result can be interpreted as follows: values ≤ 0 indicate no agreement; 0.01–0.20 none to slight; 0.21–0.40 fair; 0.41– 0.60 moderate; 0.61–0.80 substantial; and 0.81–1.00 as almost perfect agreement [25]. The survival times of patients were summarised using median and interquartile (IQ) ranges. Kaplan Meier survival curves were created to describe and compare survival across prognostic categories and disease trajectories. Patients for whom survival since prognosis was longer than a year were censored at day 400 in the Kaplan Meir curves. The log-rank test was used to compare survival between defined groups.

The characteristics of patients for whom prognosis was overestimated or underestimated were examined.

A sensitivity analysis excluding individuals who had a CMC record created after the start of the COVID-19 pandemic (March 2020) was carried out on the primary outcome i.e., the accuracy of the survival estimates to examine the hypothesis that prognostic accuracy may have been inflated, especially during the early months of the pandemic.

The study was reported using the RECORD (REporting of studies Conducted using Observational Routinely-collected Data) statement – extended from the STROBE (STrengthening the Reporting of OBservational studies in Epidemiology) statement (Supplementary Table 1).

Using a large set of routinely collected data we evaluated the accuracy of temporal predictions of survival in patients with a variety of advanced illnesses.

Our analysis demonstrates that the overall clinical accuracy of categorising patients into broad prognostic groups was 62%, mirroring prognostication estimates in other clinical studies [9, 14]. Clinicians were particularly good at identifying patients who were likely to live for fewer than 14 days. In a recent large multi-centre study of 1896 patients with cancer admitted to a palliative care unit, as in our study, survival estimates (using clinician’s predictions of survival or prognostic scales) were most accurate in patients with days to live. In that study however, all methods of prognostication demonstrated good performance for up to 60 days of survival, which is likely explained by the underlying diagnosis and the short overall median survival time (19 days), suggesting that patients were likely to have had more predictable trajectories [26]. In our study of patients with a variety of diagnoses and likely disease trajectories, clinicians were also good at identifying patients who were likely to live for more than one year but were less accurate at predicting survival of “weeks” or “months”. The greatest extent of agreement between estimated and actual prognosis was found for patients with dementia/frailty or cancer. The median survival of patients predicted to live for days, weeks, months, or years, fell within the expected range for the whole cohort and each disease category.

In this study, the accuracy of predictions about imminent death (within two weeks) was 74.2% across the full patient cohort. The best accuracy for this time frame was in cancer patients (79.5%) and the worst was in patients with respiratory disease (63.5%). Our findings align with the more typically predictable short period of rapid decline which is often described in patients with cancer, as compared to the gradual and progressive deterioration in patients with other diagnoses [20, 27, 28]. For patients with heart or lung disease, frequent exacerbations in their clinical condition on the background of a gradual decline in health and functional status make identification of the dying phase more challenging [20]. Nonetheless, our results provide reassurance that for the majority of patients, regardless of diagnosis, clinicians are reasonably accurate at identifying the last few days of life.

Clinicians were also good at recognising which patients were likely to live for more or less than one year. Of the 55,347 patients predicted to survive for more than one year, 45,920 (83%) did so. Conversely, of the 39,840 people predicted to live for less than one year, 30,726 (77%) did so.

This study demonstrates the significant difference in the accuracy of survival estimates depending on the prognostic category. Factors that influence the accuracy of survival estimates are yet to be defined but are likely to be complex and multiple [18]. Whilst prognostic models vary in levels of sophistication [29], despite its central importance, clinicians are rarely trained for prognostication [30]. Clinicians usually consider a number of patients’ characteristics when providing a prognosis, such as performance status, underlying diagnosis, clinical observations, symptoms, comorbidities and the known overall disease trajectory [24, 31‐33]. Clinicians also often rely on their own clinical experience and intuition. This may, however, be subject to explicit or implicit cognitive bias [17]. Clinicians acknowledge the challenges of estimating survival and discussing estimates with patients and families [23]. In addition, there is an inherent human desire to preserve hope, which may consciously or unconsciously affect clinicians’ communications about prognosis. Timing of prognosis is relevant, as evidence suggests that survival estimates may become more accurate when people are nearing end-of-life, a phenomenon known as “the horizon effect” [1, 34]. This finding is mirrored in the data presented in this study. However, survival predictions remain a challenge for many clinicians, even for patients who are imminently dying [35].

Our study shows that it is more difficult to identify patients who will die within “weeks” or “months” than “days” or “years”. Although many of these patients died within or earlier than the upper bound of the predicted period (e.g., most patients predicted to die within “weeks” died in fewer than 60 days [8,387/12,457 = 67%]), these data highlight the significant prognostic uncertainty faced by patients, who are neither imminently dying nor expected to live for “years”. Approaches and interventions to support patients whose situations are clinical uncertain assume greater importance [36‐38] greatly amplified during the COVID-19 pandemic [39]. Advance care planning (i.e., enabling patients to define goals and preferences for future medical treatment and care, and to record and share these discussions and decisions with relevant health and social care providers) [40] attempts to ensure that care is patient-centred even when their situation is uncertain. Early palliative care may also have a role here. The introduction of palliative care early in a patient’s disease journey has been shown to support communication and improve quality of life [3, 41, 42].

The majority of the records contained within this Electronic Palliative Care Coordination System included a prognostic estimate, which is reflective of the nature of such systems. However, in our cohort, patients with missing information on prognosis were much more likely to die in hospital and much less likely to die at home, despite the finding that a similar proportion wanted to die either at home or in a care home. Sensitive and timely discussions around prognosis may be a helpful part of the process of advance care planning for some patients, acknowledging that this is not something that every patient wants to engage with.

In this study, we observed high levels of prognostic accuracy for patients with dementia who were expected to die within days and for those with a life expectancy of year/years but poor accuracy of survival estimates of “weeks” or “months”. Patients with dementia however often experience an unpredictable disease trajectory. This can make identification of the end-of-life and prognostication challenging [43]. Our data support international recommendations that the initiation of palliative care for patients with dementia should be based on need rather than prognostication [44].

This was a large study involving patients with varying diagnoses. As a result, we were able to make robust estimates of the accuracy of survival for different groups of patients and evaluate prognostic accuracy. These data provide a real-life insight into survival predictions incorporated into individual patients’ clinical care decisions and care preferences. This study was large enough to enable stratification and comparison across major disease trajectories, facilitating detailed insight into disease-specific prognostication. Having a temporal estimate of survival as a mandatory data field ensured that prognosis information was available for 91% of the overall cohort in the observed period.

Our analyses were limited due to the format of the available data. The survival prediction was based on categorical values, consequently we are unable to calculate the area under the receiver operating characteristics curve (AUROC). Linear weighted kappa coefficients were used to determine “true” agreement between estimates and actual survival, considering the agreement that would be expected by chance [45] and giving an average of the kappa coefficients across the different survival categories [46]. Our use of LWK and percentage of accurate estimates supports comparison with other prognostic studies in which this approach was used [9, 18].

Patients with a Coordinate My Care record are expected to have a limited life expectancy as the service is designed for patients with advanced conditions and therefore our results cannot be generalised to other patient cohorts. This cohort was based in London, so results may not be generalisable to non-urban settings. Although the role of the clinician in documenting prognosis in Coordinate My Care is recorded, it is not possible to determine which health care professional made the prognosis or the role of the multi-disciplinary team in this process [47]. A previous systematic review concluded that there was no difference in the accuracy of prognostic estimates between clinician sub-groups [18]. Prognostic estimates may be revised over time, and subsequent estimates may be more or less accurate than initial ones. We did not include longitudinal changes in prognostication in our analysis. Our data included some Coordinate My Care records that were created during the initial months of the COVID-19 pandemic. It is possible that the onset of a new and unknown disease made accurate prognoses even more difficult, Our data did not however demonstrate any difference in prognostic accuracy when prognostic estimates from records created after the start of the pandemic were included.

Our study was based on the retrospective analysis of routinely collected data and does not give any information about the information needs of individual patients about prognostic estimates.