The agreement between chronic diseases reported by patients and derived from administrative data in patients undergoing joint arthroplasty

The study sample of 676,428 patients was drawn from patient-reported data collected by the national PROMs programme in the English NHS [12]. All hospitals providing elective hip or knee arthroplasty funded by the English NHS are required to participate and patients are asked to complete pre-operative and post-operative questionnaires about their hip or knee condition and general health.

The data sample comprised completed pre-operative questionnaires linked with routinely collected administrative hospital data, Hospital Episode Statistics (HES) data, on all patient who had a hip or knee arthroplasty carried out in the English NHS between April 2009 and March 2016. The HES database contains a record of every inpatient hospital admission in the English NHS and is used primarily for reimbursement purposes [13]. A linked pre-operative PROMs questionnaire and HES record is available for 71% of eligible hip and knee arthroplasties [12].

We created a dataset comprising one unique linked patient-reported record for each individual patient. Duplicate records were excluded if more than one pre-operative questionnaire was linked to a procedure or more than one procedure in HES was linked to the same questionnaire. The first linked HES record for each patient was included but linked records for any subsequent procedures were excluded. Patients were also excluded if they reported seven or more comorbidities in the preoperative PROMs questionnaire due to the concerns about the validity of the responses. Patients appeared to report the absence rather than the presence of a chronic disease.

In the PROMs pre-operative questionnaire patients were asked: ‘Have you ever been told by a doctor that you have any of the following conditions? Heart disease (for example angina, heart attack or heart failure), high blood pressure, problems caused by stroke, leg pain when walking due to poor circulation, lung disease (for example asthma, chronic bronchitis or emphysema), diabetes, kidney disease, diseases of the nervous system (for example, Parkinson’s disease or multiple sclerosis, liver disease, cancer (within the last 5 years), depression, [or] arthritis’. ‘Arthritis’ was excluded from our analyses because it is the primary a reason for hip or knee arthroplasty (81% patients reported having arthritis).

The 11 patient-reported chronic diseases were identified within HES data using International Classification of Disease (ICD-10) codes from the corresponding linked HES record of the hip or knee arthroplasty and from HES records of any other hospital admission within the previous 12 months or five years. Each HES record includes up to 20 ICD-10 diagnosis codes.

The initial set of ICD-10 codes for each of the 11 chronic diseases was derived from three chronic disease indices that have been used to identify chronic diseases in administrative data: The Royal College of Surgeons of England Charlson Comorbidity Index (RCS CCI) [14], the Quan Charlson Comorbidity Index (Quan CCI) [15] and the Elixhauser Comorbidity index [16]. The RCS CCI was chosen because it was designed to predict outcomes in surgical patients and has been validated for total hip arthroplasty using English HES data [14]. The Quan CCI is an adaptation of the Deyo CCI [15], and was chosen because it uses ICD-10 coding and is similar to other CCI adaptations in predicting both short-term and long-term mortality [17]. The Elixhauser Comorbidity Index was chosen because there is evidence that it may predict mortality better than other adaptations of the CCI [18].

The set of ICD-10 codes derived from the three chosen comorbidity indices were then mapped to the 11 diseases included in the PROMs questionnaire (see Additional file 1). A further 16 ICD-10 codes were added to the chronic disease mapping through the process of ‘backward coding’. ‘Backward coding’ involved reviewing linked HES records of hospital admissions in patients who had reported a chronic disease but who had no mapped records (ICD-10 codes) of the chronic disease in their HES records. First, relevant ICD-10 chapters were identified for each of the 11 chronic diseases. The most common (> 1% of patients reporting the chronic disease) and clinically relevant codes at the ICD-10 three-character category level were then identified. Second, the codes identified at the ICD-10 three-character level were further investigated at the ICD-10 four-character subcategory level. The prevalence of each four-character code in the administrative data was compared between patients who had and those who had not reported a specific chronic disease. The four-character code was added to the mapped ICD-10 codes if the proportion of patients reporting presence of a chronic disease was at least twice that in patients not reporting the chronic disease. For the main analyses, this final set of codes was used to determine the presence of chronic disease according to administrative data from the corresponding linked hospital record and from records of admissions within the previous 12 months or five years.

The patient-reported chronic diseases at the point of surgery were compared with recorded diagnoses in the corresponding administrative record of the linked hospital admission and the records of previous admissions in two ways. First, agreement between patient-reported and administrative records was evaluated using sensitivity and specificity with administrative data as the reference standard. Second, we calculated the kappa statistic as an alternative measure of the agreement between patient-reported and administrative data for each condition. The kappa statistic is an agreement measure that takes into account chance agreement. A value of one indicates perfect agreement and a value of zero indicates no agreement above that expected by chance. Kappa values are often categorised in the following way: < 0.40 ‘poor agreement’, 0.40–0.60 ‘moderate agreement’, 0.61–0.80 ‘substantial agreement’, and 0.81–1.00 ‘near perfect agreement’ [19].

The sensitivity of patient-reported chronic disease was also explored further at the chronic disease subcategory level derived from administrative data. We grouped the set of ICD-10 codes for each of the 11 comorbid conditions according to clinically relevant subcategories (see Additional file 1). ICD-10 codes were grouped according to whether they reflected a cause (e.g. subarachnoid haemorrhage), a manifestation (e.g. asthma), or a consequences of disease (e.g. renal failure). For each comorbid condition ICD-10 codes that did not fit into any these grouping, the codes were put into an ‘other’ group. The sensitivity of the patient-reported chronic diseases compared to these chronic disease subcategories derived from administrative data was presented in a forest plot.

The impact of the length of the look-back period on the performance of the combined chronic disease measure in administrative data was also investigated [20‐22]. Some chronic diseases such as ‘heart disease’ are diseases that can fluctuate and others, such as ‘stroke’ are single events. For that type of chronic diseases, a longer look-back ensures that records of chronic diseases coded in admissions that occurred further in the past are also captured. In the PROMs questionnaire, patients were asked to recall cancer within the last five years which is another reason to use a 5-year look-back period as an alternative to the one-year look-back period.

Agreement between chronic disease measures reported by patients and derived from administrative data was examined in 676,428 patients who underwent a hip or knee arthroplasty between 2009 and 2016 in the English NHS and who participated in the PROMs programme from a total 791,369 linked records. Records were excluded for the following reasons: duplicate pre-operative questionnaires (10,762), duplicate HES procedures (140), subsequent procedures for patients included in the analyses (103,395), and patients reporting seven or more chronic diseases on their pre-operative questionnaire (644) (see Fig. 1). 50.6% of the patients underwent knee arthroplasty. The average age of the population was 68 years (18–105). The majority of the patients had a white ethnic background (86.3%) and 58.0% of the study cohort were female (see Table 1). Patients living in the most socioeconomically deprived areas were slightly under-represented in the sample: those in the bottom two deprivation groups based on quintiles made up only 34.5% of patients undergoing a primary hip or knee arthroplasty whereas 40% is expected given that the quintiles reflect the national distribution.

Sensitivity, specificity, and the kappa statistic for patient-reported chronic disease against chronic diseases derived from administrative data using a 1-year look-back are reported in Table 1. Patient-reported chronic diseases had high specificity (ranging between 90.3% for ‘high blood pressure’ and 99.7% for ‘disease of the nervous system’ and ‘liver disease’), but sensitivity varied (ranging from 18.8% for ‘kidney disease’ to 87.5% for ‘diabetes’).

According to the kappa statistic, there was ‘substantial agreement’ between patient-reported and administrative results for ‘high blood pressure’ (κ =0.65) and ‘almost perfect agreement’ for ‘diabetes’ (κ =0.88) (see Table 2). There was ‘moderate agreement’ for ‘heart disease’ (κ =0.54) and ‘lung disease’ (κ =0.55). In contrast, there was ‘poor agreement’ for ‘stroke’, ‘liver disease’, ‘leg pain due to poor circulation’, ‘kidney disease’ and ‘depression’. Agreement between patient-reported chronic diseases and chronic disease subcategories derived from administrative data.

Further investigation comparing patient-reported chronic disease against chronic disease subcategories derived from administrative data demonstrated that the sensitivity varied if the patient-reported results were compared against subcategories defined according to administrative data (see Fig. 2). Sensitivity ranged from 1.3% for patient-reported ‘leg pain due to poor circulation’ compared against ‘gangrene’ according to administrative data to 91.6% for patient-reported ‘diabetes’ compared against ‘insulin-dependent diabetes’ according to administrative data.

The sensitivity was considerably higher in subgroups of chronic diseases where specific examples of the chronic diseases were given as examples in the questionnaire used for the PROMs survey in the PROMs survey. For example, we saw that the sensitivity of ‘diseases of the nervous system (for example Parkinson’s disease or multiple sclerosis)’ was much higher in subgroups of patients who had these two specific diseases quotes as examples in their administrative data (65.6 and 69.5%, respectively) than in entire group of patients who had the generic term ‘diseases of the nervous system’ in the administrative data (20.9%). We saw a similar effect for the examples given in ‘heart disease (for example angina, heart attack or heart failure)’. The sensitivity in the subgroup of patients with the specific term ischemic heart disease in the administrative data was significantly higher (64.9%) than in all patients who had the generic term ‘heart disease’ according to administrative data (46.4%).

The impact of the length of look-back period on the chronic diseases derived from administrative data was investigated. Increasing the look-back period for identifying chronic diseases in administrative data from 12 months to five years had little impact on the sensitivity, specificity and kappa statistic (see Table 3). As expected, sensitivity decreased and specificity increased. The biggest change was the increase of the kappa statistic for ‘cancer’ from 0.37 with a 12-month look-back period to 0.69 with a 5-year look-back period.