Background
Objective
Methods
Stage 1: Indicator development
Stage 2: Indicator testing
Attribute | Summary of method |
---|---|
Clarity | • RAND Appropriateness Method ratings * |
Necessity | • RAND Appropriateness Method ratings * |
Acceptability | • Risks, issues, relative impact, and uncertainties (interviews with practice staff) |
Feasibility | • 'Technically feasiblility' in current family practice systems and whether supported by current methods of data extraction for QOF (data extraction in all family practice clinical systems) |
Reliability | • Reproducible in testing (data extraction: test-retest) |
Implementation | • Baseline and potential change in baseline;evidence of sensitivity to change (data extraction); |
• Exception reporting/gaming (interviews with practice staff); | |
• Changes in practice organisation; potential barriers; workload (interviews with practice staff and workload diaries) | |
• Unintended consequences (interviews with practice staff) | |
Changes to any existing QOF indicators | • Summary of any suggested changes to existing QOF indicators as a result of piloted indicators |
Changes in wording of the indicator(s) | • Summary of any suggested changes to indicator wording |
Cost effectiveness | • Summary of evidence of cost effectiveness (cost-effectiveness modelling) |
Overall recommendation
| 1) no major barriers/risks/issues/uncertainties |
2) some barriers/risks/issues/uncertainties but okay | |
3) major barriers/risks/issues/uncertainties preclude it |
Clarity and necessity: content validity
Technical feasibility and reliability
Implementation - workload
Implementation: sensitivity to change and discrimination/variance
Acceptability and implementation
Cost effectiveness
Indicator piloting: data collection methods and analyses
Overall recommendations
Results
Clarity and necessity: content validity
Clinical area | Indicator wording | Implementation issues raised as a result of piloting | Recommendations | |
---|---|---|---|---|
1 | Asthma | The percentage of patients with asthma who have had an asthma review in the previous 15 months that includes an assessment of asthma control using the 3 Royal College of Physicians questions (control of daytime, night time and activity limiting symptoms in the last week. | • Technical feasibility: need for revised IT data entry templates | Acceptability: no barriers |
• Administration of questions by phone, post or face to face | Feasibility: some barriers | |||
Reliability: some barriers | ||||
Implementation: no barriers | ||||
Overall: some barriers to address | ||||
2 | Dementia | The percentage of patients with a new diagnosis of dementia to have FBC, calcium, glucose, renal and liver function, thyroid function tests, serum vitamin B12 and folate levels recorded 6 months before or after entering on to the register | • Inconsistency with dementia referral pathways/referral governance | Acceptability: no barriers |
• Variation in practice - QOF danger of standardizing practice | Feasibility: no barriers | |||
• Low prevalence in some practices (as confirmed by the pilot) | Reliability: no barriers | |||
• Tests of calcium levels not always routinely done | Implementation: some barriers | |||
Overall: some barriers to address but can go forward | ||||
3 | Diabetes | The percentage of patients with diabetes with a record of testing of foot sensation using a 10 g monofilament or vibration (using biothesiometer or calibrated tuning fork), within the preceding 15 months | • Changes required to data entry diabetes templates where not already included | Acceptability: no barriers |
• Education or training of nurses in some practices. | Feasibility: no barriers | |||
Reliability: no barriers | ||||
Implementation: no barriers | ||||
Overall: no barriers | ||||
4 | Diabetes | The percentage of patients with diabetes with a record of a foot examination and risk classification: 1) low risk (normal sensation, palpable pulses), 2) increased risk (neuropathy or absent pulses), 3) high risk (neuropathy or absent pulses plus deformity or skin changes or previous ulcer) or 4) ulcerated foot within the preceding 15 months | • Changes required to data entry diabetes templates where not already included | Acceptability: no barriers |
• Education or training of nurses in some practices. | Feasibility: no barriers | |||
• Routine data recording | Reliability: no barriers | |||
• Workload implications for practice staff | Implementation: some barriers | |||
• Attribution problems in terms of payments attached to QOF if performed outside the practice | Overall: some barriers to address but can go forward | |||
• Time taken to perform | ||||
5 | Myocardial Infarction | The percentage of patients with a history of myocardial infarction (from 1 April 2011 {from 1 October 2009 for the purposes of piloting} currently treated with an ACE inhibitor, aspirin or an alternative anti-platelet therapy, beta-blocker and statin (unless a contraindication or side effects are recorded) | Implementation issues: | Acceptability: some barriers |
• Problems of attribution/hospital led prescribing | Feasibility: no barriers | |||
• PCT formulas/guidance/pharmacy advisors content and advice is contrary to NICE guidelines | Reliability: no barriers | |||
MI was the only area for which there is useable cost-effectiveness data | Implementation: some barriers | |||
Overall: some barriers to address but can go forward | ||||
6 | Myocardial Infarction | The percentage of patients with a history of myocardial infarction who have a record of intolerance or allergy to an ACE inhibitor who are currently treated with an ARB (unless a contraindication or side effects are recorded) | • Conflicts with variations in local guidance (in some areas PCT stipulates trying 3 ACEs before an ARB, in others ARB is front line treatment) | Acceptability: some barriers |
• Variation in percentage of patients on ARB | Feasibility: no barriers | |||
• Problems of attribution/hospital led prescribing | Reliability: no barriers | |||
• Variations in local procedures in i.e. PCT formulas/guidance/pharmacy advisors | Implementation: some barriers | |||
• Private initiated prescribing effects % of patients on ARB | Overall: some barriers to address but can go forward | |||
MI is the only area for which there is useable cost-effectiveness data | ||||
7 | Serious Mental Illness | The percentage of patients with schizophrenia, bipolar affective disorder and other psychoses who have a record of alcohol consumption in the preceding 15 months | • Seldom heard group, whom often consult opportunistically. Difficulty getting patients to come back in | Acceptability: some barriers |
Feasibility: no barriers | ||||
Reliability: no barriers | ||||
Implementation: some barriers | ||||
8 | Serious Mental Illness | The percentage of patients with schizophrenia, bipolar affective disorder and other psychoses who have a record of BMI in the preceding 15 months | • Low levels of cervical screening | Overall: some barriers to address but can go forward |
9 | Serious Mental Illness | The percentage of patients with schizophrenia, bipolar affective disorder and other psychoses who have a record of blood pressure in the preceding 15 months | • Perceived clinical irrelevance to younger patients re. annual cholesterol, BP, HBA1c/glucose check: these are not routinely done in all practices for all patients on the current register | |
10 | Serious Mental Illness | The percentage of patients with schizophrenia, bipolar affective disorder and other psychoses who have a record of total cholesterol: hdl ratio level in the preceding 15 months | • May create undue focus on individual processes/indicators in unbundled indicators rather than the overall physical health of patient "appropriate to their age, gender and health status" | |
11 | Serious Mental Illness | The percentage of patients with schizophrenia, bipolar affective disorder and other psychoses who have a record of blood glucose level or HBA1c in the preceding 15 months | ||
12 | Serious Mental Illness | The percentage of women aged 25-64 with schizophrenia, bipolar affective disorder and other psychoses who have a record of cervical screening within the last 5 years | ||
13 | Palliative Care | The percentage of patients on the palliative care register who have a preferred place to receive end-of-life care documented in the records | • Perceived potential harm to patients | Acceptability: preclude |
• Changes to timing of which patients are put on the register: the palliative care register is perceived to often be quite subjective | Feasibility: no barriers | |||
• The indicator does not pro-actively encourage GPs to keep the preferred place for end of care up to date | Reliability: no barriers | |||
• Anxiety over the rigidity of the stipulated timeframes which are too prescriptive | Implementation: preclude | |||
• Undue focus on one isolated question from a mutlifaceted and complex issue | Overall: preclude | |||
• Problems of attribution/not necessarily general practice team's responsibility |
Technical feasibility and reliability
Implementation - workload
Implementation: sensitivity to change and discrimination/variance
Acceptability and implementation
Issues raised by piloting
Cost effectiveness
Overall recommendations
Discussion
Strengths and limitations of the testing protocol
Strength
Limitations
Implications for practice and policy
UK-ITP+
| AMA-PCIP [10] | NCQA/HEDIS [6] | Qualify [7] |
---|---|---|---|
NICE prioritisation | Needs assessment | Relevance | Relevance |
Acceptability | Acceptability | ||
Clarity | Clarity | Precision = | Clarity/specificity |
Feasibility | Feasibility | Feasibility | Feasibility |
Reliability | Reliability | Reproducibility~ | Reliability~ |
Validity * | Validity *# | Validity * | Validity *~ |
Implementation | Implementation | Implementation | Implementation |
-Unintended | -Unintended | -Controllability¬ | -Risks/side effects¬ |
Consequences | Consequences | -Benchmarking/ | -Benchmarking/ |
-Benchmarking/ | -Benchmarking | ||
Sensitivity to change -Workload | |||
Cost effectiveness | Cost effectiveness¬ |
Conclusions
Appendix 1: Summary of indicator testing protocol
Clarity
Content validity
-
Each indicator is underpinned by a NICE/SIGN guideline.
-
Each indicator is rated necessary by a RAND Appropriateness Method, which is based on physicians/practice staff adhering to the indicator providing a higher quality of care/service than those who are not doing so.