Background
Over the course of the last twenty years, doctors have become used to reviewing the quality of their clinical work. We are in an era of performance management, where the quality of doctors' work is judged by external bodies, and doctors are being offered financial incentives to achieve quality and other targets set by commissioners of services. The new general medical services contract, covering the work of general practitioners (GPs) in the UK National Health Service (NHS), includes clinical performance targets in the GPs' employment contracts. GPs' remuneration is now linked directly to the attainment of targets contained in a set of indicators called the Quality and Outcomes Framework (QOF) [
1].
The QOF indicators are divided into four domains: clinical, organisational, patient experience, and additional services [
2,
3] (See Table
1 for a summary of the domains and their indicators). The clinical domain indicators apply to clinical or clinically driven processes and are arranged into disease specific groups. Some measure the extent of particular interventions, such as the proportion of diagnosed coronary heart disease (CHD) patients taking aspirin, an alternative antiplatelet therapy, or an anti-coagulant. Often, the indicators cover the administrative tasks which support effective clinical management, such as maintenance of disease registers. The organisational domain is designed to reflect the quality of basic administrative processes, the patient experience domain largely measures processes around the administration of the annual patient survey, and the additional services domain covers a range of specific additional services offered by some practices.
Table 1
Summary of the indicators included in each QOF domain
Cervical screening |
Child health surveillance |
Contraception |
Maternity services |
Clinical domain
|
Asthma |
Atrial fibrillation |
Cancer |
Chronic kidney disease |
Chronic obstructive pulmonary disease |
Dementia |
Depression |
Diabetes mellitus |
Epilepsy |
Heart failure |
Hypertension |
Hypothyroidism |
Learning difficulties |
Mental health |
Obesity |
Palliative care |
Secondary prevention in coronary heart disease |
Smoking indicators |
Stroke & transient ischaemic attacks |
Organisation domain
|
Education & training |
Information for patients |
Medicines management |
Practice management |
Records & information about patients |
Patient experience domain
|
Length of consultation |
Patients surveys |
The success or failure of this target-based approach depends in significant part on two issues, namely whether it is possible to design a robust and clinically meaningful set of indicators, and whether financial incentives change doctors' behaviour and thereby improve patient care. The aim of this study is to address the first of these issues, by assessing the extent to which the UK government's new QOF scores are correlated with measures of health observed in practice populations, after accounting for well-established associations between health outcomes and demographic and socio-economic factors. The outcome measures considered were those that were readily obtainable from routinely collected data, namely mortality and hospital admission.
Results
In PCT 1, 14.3% of the population were aged over 65 years compared to 17.4% in PCT 2. Males made up 50.3% and 49.2% of the populations of PCT 1 and 2 respectively. As a whole, PCT 1 was more deprived than PCT 2 (median income domain score of 0.18 versus 0.10). The median clinical domain scores were 531.5 in PCT 1 and 545.9 in PCT 2, the median organisational domain scores were 180 in PCT 1 and 176 in PCT 2, and the median additional services domain scores were 35.8 in PCT 1 and 36.0 in PCT 2.
Hospital admissions
Tables
3 and
4 show the results of the models for emergency admissions for the two PCTs. Higher additional services domain scores were significantly associated with increased admissions for asthma, CHD, stroke and other conditions in PCT 1 and cancer in PCT 2. Increasing clinical domain scores were generally associated with reduced admissions, although this was only significant for cancer and other conditions in PCT 2 (OR 0.86; 95% CI 0.79–0.93 and OR 0.94; 95% CI 0.92–0.97 per quartile increase in score respectively). There was no clear association between the organisational domain and hospital admissions.
Table 3
Associations between patient, area and GP level covariates and hospital admissions and mortality in PCT 1
Patient-level; OR (95% CI)
| | | | |
Female | 1.42 (1.17–1.75) | 0.68 (0.57–0.80) | 0.90 (0.76–1.07) | 0.56 (0.48–0.64) |
5-year age group | 1.01 (0.99–1.03) | 1.38 (1.34–1.41) | 1.45 (1.42–1.49) | 1.41 (1.38–1.44) |
Area-level; OR (95% CI)
| | | | |
Income domain* | 1.23 (1.14–1.32) | 1.07 (1.00–1.14) | 1.38 (1.29–1.48) | 1.15 (1.08–1.22) |
GP-level; OR (95% CI)
| | | | |
Additional Services* | 1.04 (1.01–1.08) | 1.00 (0.97–1.03) | 1.02 (0.98–1.05) | 1.03 (1.01–1.07) |
Clinical* | 1.05 (0.96–1.14) | 0.99 (0.92–1.05) | 0.99 (0.92–1.07) | 0.97 (0.91–1.03) |
Organisational* | 0.95 (0.89–1.03) | 1.02 (0.95–1.09) | 0.98 (0.91–1.05) | 0.98 (0.92–1.03) |
Variance (95% CI)
| | | | |
Area-level | 0.00 (0.00–0.03) | 0.00 (0.00–0.02) | 0.11 (0.00–0.19) | 0.04 (0.01–0.11) |
GP-level | 0.01 (0.00–0.05) | 0.01 (0.00–0.07) | 0.02 (0.00–0.08) | 0.01 (0.00–0.05) |
|
Emergency admission
|
Mortality
|
|
Diabetes
|
Stroke
|
Other
|
All causes
|
|
(N = 164)
|
(N = 360)
|
(N = 18,784)
|
(N = 3,201)
|
Patient-level; OR (95% CI)
| | | | |
Female | 0.81 (0.60–1.12) | 0.91 (0.74–1.12) | 1.10 (1.06–1.13) | 0.69 (0.64–0.75) |
5-year age group | 1.13 (1.09–1.17) | 1.55 (1.49–1.61) | 1.14 (1.13–1.14) | 1.68 (1.65–1.70) |
Area-level; OR (95% CI)
| | | | |
Income domain* | 1.21 (1.08–1.36) | 1.10 (1.01–1.19) | 1.14 (1.12–1.16) | 1.10 (1.06–1.14) |
GP-level; OR (95% CI)
| | | | |
Additional Services* | 1.03 (0.98–1.10) | 1.05 (1.01–1.11) | 1.03 (1.01–1.04) | 1.01 (0.99–1.03) |
Clinical* | 0.94 (0.83–1.05) | 1.00 (0.92–1.10) | 0.99 (0.96–1.03) | 0.97 (0.92–1.01) |
Organisational* | 1.03 (0.92–1.16) | 0.96 (0.89–1.04) | 0.98 (0.95–1.02) | 1.02 (0.98–1.07) |
Variance (95% CI)
| | | | |
Area-level | 0.01 (0.00–0.21) | 0.00 (0.00–0.02) | 0.01 (0.00–0.01) | 0.05 (0.03–0.07) |
GP-level | 0.02 (0.00–0.16) | 0.01 (0.00–0.04) | 0.04 (0.03–0.06) | 0.03 (0.01–0.06) |
Table 4
Associations between patient, area and GP level covariates and hospital admissions and mortality in PCT 2
Patient-level; OR (95% CI)
| | | | |
Female | 1.96 (0.99–4.17) | 0.74 (0.57–0.94) | 0.92 (0.70–1.23) | 0.54 (0.43–0.68) |
5-year age group | 0.92 (0.85–1.00) | 1.38 (1.33–1.44) | 1.43 (1.37–1.50) | 1.38 (1.34–1.43) |
Area-level; OR (95% CI)
| | | | |
Income domain* | 1.27 (1.00–1.58) | 1.14 (1.03–1.25) | 1.42 (1.28–1.57) | 1.16 (1.07–1.26) |
GP-level; OR (95% CI)
| | | | |
Additional Services* | 0.99 (0.96–1.04) | 1.03 (1.01–1.05) | 1.00 (0.98–1.03) | 1.00 (0.98–1.01) |
Clinical* | 0.87 (0.71–1.06) | 0.86 (0.79–0.93) | 0.94 (0.86–1.02) | 1.00 (0.93–1.09) |
Organisational* | 0.96 (0.90–1.04) | 1.00 (0.97–1.04) | 1.05 (1.01–1.09) | 1.01 (0.98–1.04) |
Variance (95% CI)
| | | | |
Area-level | 0.04 (0.00–0.58) | 0.00 (0.00–0.12) | 0.01 (0.00–0.06) | 0.01 (0.00–0.04) |
GP-level | 0.02 (0.00–0.87) | 0.01 (0.00–0.17) | 0.01 (0.00–0.13) | 0.01 (0.00–0.06) |
|
Emergency admission
|
Mortality
|
|
Diabetes
|
Stroke
|
Other
|
All causes
|
|
(N = 58)
|
(N = 160)
|
(N = 6,847)
|
(N = 1,423)
|
Patient-level; OR (95% CI)
| | | | |
Female | 0.72 (0.42–1.22) | 0.85 (0.62–1.16) | 1.10 (1.05–1.16) | 0.71 (0.64–0.80) |
5-year age group | 1.15 (1.09–1.23) | 1.62 (1.52–1.72) | 1.12 (1.11–1.12) | 1.69 (1.66–1.73) |
Area-level; OR (95% CI)
| | | | |
Income domain* | 1.15 (0.95–1.38) | 1.17 (1.03–1.31) | 1.13 (1.10–1.16) | 1.11 (1.06–1.17) |
GP-level; OR (95% CI)
| | | | |
Additional Services* | 0.99 (0.96–1.02) | 1.02 (1.00–1.05) | 1.01 (1.00–1.01) | 1.01 (0.99–1.02) |
Clinical* | 1.11 (0.92–1.37) | 0.95 (0.86–1.07) | 0.94 (0.92–0.97) | 0.98 (0.92–1.05) |
Organisational* | 0.98 (0.92–1.06) | 0.99 (0.95–1.04) | 1.00 (1.00–1.01) | 0.99 (0.97–1.01) |
Variance (95% CI)
| | | | |
Area-level | 0.02 (0.00–0.11) | 0.01 (0.00–0.08) | 0.01 (0.00–0.02) | 0.02 (0.00–0.06) |
GP-level | 0.02 (0.00–0.44) | 0.02 (0.00–0.19) | 0.01 (0.00–0.02) | 0.04 (0.01–0.13) |
Within both PCTs, females had significantly fewer admissions than males for cancer and CHD, but females were more likely to be admitted for asthma and other conditions. Age was positively associated with all admissions in both PCTs, except those for asthma. Higher deprivation scores were associated with an increased likelihood of admission for all conditions. Those most impacted upon by deprivation were COPD (PCT 1 OR 1.38; 95% CI 1.29–1.48 and PCT 2 OR 1.42; 95% CI 1.28–1.57 per quartile increase in income domain score), and asthma (PCT 1 OR 1.23; 95% CI 1.14–1.32 and PCT 2 OR 1.27; 95% CI 1.00–1.58 per quartile increase).
The area-level and GP-level variance terms refer to the amount of unexplained variation at each of these levels. A figure of zero, or close to zero, means that there is little or no residual variation, i.e. all the variation at that level has been explained. Generally, the variances were small, both at the area-level and GP-level, meaning that there is little unexplained variation at both levels.
All cause mortality
None of the QOF variables were significantly associated with mortality in either PCT. As expected, increasing age was significantly associated with increased mortality, whilst being female was significantly associated with decreased mortality (in both PCTs). Increasing deprivation was associated with significantly increased mortality (PCT 1 OR 1.10, 95% CI 1.06–1.14 and PCT 2 OR 1.11, 95% CI 1.06–1.17 per quartile increase in income domain score). As with emergency admissions, there was little unexplained variation in mortality at the area- and GP-levels.
Discussion
This study provides an early insight into the relationship between QOF scores and patient outcomes measured in terms of emergency hospital admissions and mortality. Whilst there were several associations between QOF and the outcome measures, these were not consistent. Generally, the results suggest a greater association between the outcomes and deprivation than with any of the QOF scores. However, when significant associations between deprivation and the outcome measures were observed, there was still some unexplained variation at the SOA level, suggestive of unmodelled factors operating within the patient (area-level) environment. These include levels of crime, education and unemployment and the availability of leisure facilities. A study in Canada found that health correlated more frequently with non-medical determinants (i.e. health behaviours and living and working conditions) than healthcare performance indicators [
8].
Looking in more detail at the QOF measures, higher clinical domain scores were generally associated with lower odds of unplanned admission. Activities such as the maintenance of disease registers, high levels of prescribing and opportunistic screening may be driving this in high scoring practices. However, the associations were not consistent and were significant only for cancer and other conditions (in PCT 2). The fact that performance indicators are associated with small effects on outcomes has been reported previously [
9]. Unsurprisingly, there was no clear association with outcomes for the domain that has less obvious relevance to admissions and mortality, the organisational domain. There was an association between higher scores in the additional services domain and increased odds of unplanned admission for a number of disease groups. The additional services covered in the indicator set are largely aimed at younger patient groups (e.g. child health surveillance, emergency contraception, maternity services), whereas, the outcomes captured in this study (admission and death) are concentrated amongst older people. It is possible that higher levels of additional services lead to a resource/time competition between the care of young people and care of the elderly and to higher admissions/mortality in the elderly. However, this is speculation and we are unable to explain the increase in admissions for specific disease groups. Mortality was associated with the socio-demographic variables in line with expectation. However, mortality showed no significant relationship with any of the QOF domains. Whilst both admission and mortality will be associated with long-term factors, of the two, admission rates may be more likely to be modified by short term interventions than mortality.
We should not be surprised by some lack of consistency in the associations found in this study. The complexity of the relationship between processes and outcomes in healthcare has been known for some time, as illustrated by Donabedian in 1966 [
10]. Perhaps, as an acknowledgement of the complexity with which variables operate in community settings, the author of this early work specifically excluded community care delivery from the scope of the paper. More recent work on how different kinds of indicator operate has provided a critique of quality assessment based upon process indicators [
11]. This work highlights the extent to which process variables are confounded by the effects of factors such as deprivation and case mix. This supports the use of a hierarchical approach for investigating the relationships between process and outcome whilst adjusting for confounders operating at different levels.
Few studies using QOF data have been published, although those that have suggest that the current indicators do not adequately measure quality of care. One study looked at the correlation between six performance indicators, one of which was the QOF, for the 303 PCTs in England [
12]. They found little correlation, except in the specific area of screening, and suggested that current indicators do not have sufficient construct validity to measure the underlying concept of quality. Another more focused study investigated whether higher QOF scores were associated with better stroke care, assessed using the Royal College of Physicians (RCP) guidelines [
13]. Higher stroke quality scores did not reflect better adherence to RCP guidance and it was concluded that further research is needed to assess the generalisability of this finding and how the QOF might be better aligned with delivering best practice.
The distribution of the scores in all domains was clustered towards the upper limit of their maximum, which leads us to question to what extent the QOF scores are a sensitive measure of quality of care. It may be that disaggregating the scores and focusing upon some of the higher level measures within the domains is preferable. Another option would be to use percentage achievement. Indicators for which the targets are more difficult to achieve tend to have lower maximum thresholds and so more variation may be found using percentage achievement than by the points awarded.
A recent article by Guthrie and colleagues suggests that the value of the QOF could be improved by incorporating treatment data in to the quality indicators, as the current targets fail to identify clear opportunities for improving healthcare, thereby leading to therapeutic inertia [
14]. A practice may be able to achieve maximum QOF payment, for example by fulfilling the criteria of a certain proportion of patients having a blood pressure below a certain level, despite missed opportunities for tighter control such as the prescribing of new antihypertensive drugs or increasing the treatment dose where appropriate.
Strengths and limitations
The key question underpinning this work is whether or not the processes that the QOF scores measure will result in measurable changes in the populations' health. The results suggest that this does not appear to be the case. The main limitation of this study is the short lag time between the implementation of the QOF and measurement of the health outcomes, which may not be sufficient to assess the full impact of the QOF. However, the risks of hospital admission or death are a function of factors that have been operating over a period of time. Whilst most of these factors operate at the individual level (such as those related to lifestyle), some will have impacted at the practice level (such as the propensity to refer for secondary interventions, for example coronary artery by-pass operation). It will be some time before the longer-term impact of the QOF can be measured.
One feature of the QOF design is that it allows GPs to exclude certain patients from the performance calculations for specific indicators (exception reporting), for example those with terminal illness, but there is a possibility that GPs may inappropriately exclude patients where they have failed to meet the targets in order to artificially improve their performance. This may weaken any associations between the QOF scores (based on a selected population) and the health of the responsible population (without selection). Doran and colleagues looked at this and found a significant positive relationship between rates of exception reporting and reported achievement for certain clinical indicators, although, the effect was small with a 1% increase in the proportion of excluded patients leading to a 0.31% increase in reported achievement [
15]. McLean and colleagues found that the exception reporting system contributed to a continuation of the inverse care law [
16]. For quality indicators based on care delivered to all patients there was no evidence of worse care in deprived areas. However, for indicators allowing the exclusion of patients 17 of the 33 measures showed markedly worse care for more deprived populations. It is unclear how exception reporting affects the results of our study.
Due to the novelty of these indicators, data from only two PCTs were available for this initial exploration of how the target and indicator approach contributes to improving and maintaining the quality of patient care. Ideally, one would wish to examine many more PCTs to establish if consistent patterns of association exist. Also, it must be stated that the PCTs included in this study were more deprived than the "average" PCT in England. When compared against the Healthcare Commission's list of PCTs ranked by Index of Multiple Deprivation score, PCT 1 would fall in decile 8 and PCT 2 in decile 6 (where decile 10 represents the most deprived PCTs in the country). A question for further study would be whether the patterns of association between the QOF scores and the outcomes differ by level of deprivation.
Conclusion
The results of this study show that the associations between practice QOF scores and health outcomes were small and inconsistent, whilst the relationship between socio-economic deprivation and health was much stronger. This has implications for the use of target-based remuneration of GPs and other doctors, because it suggests that the assumptions about process-outcome links underpinning such schemes may be flawed. Despite some limitations, these results highlight the point that the clinical value of the QOF is dependent on the selection of the indicators, and at present, this process does not seem to have generated as sensitive a set of indicators as intended. The importance of socio-economic deprivation in explaining theobserved variation in outcomes highlights the key role of public health and health promotion, and emphasises the need to tackle inequalities and improve the health of disadvantaged groups and the population as a whole.
Competing interests
The author(s) declare that they have no competing interests.
Authors' contributions
MSG and GR designed the study. AD undertook the data linkage and managed the data. YC, YKT, and AD analysed the data. MSG, AD and GR wrote the first draft of the paper. All authors contributed to the final draft.