Quality assurance in psychiatry: quality indicators and guideline implementation

In general, quality is a theoretical construct. Psychiatric interventions take place in complex bio-psycho-social interactions and settings that are not very well understood [10]. Various efforts have been undertaken to find valid parameters of the quality of care including the structural attributes of the settings in which care occurs, the processes of care, and the outcomes of care [8]. Before assessing quality, one has to decide how quality of care should be defined and this depends on whether one assesses only provider performance or also the contributions of patients and of the health care system. Quality assessment may differ according to how broadly health and responsibility for health are defined; whether the maximum effective or average care is sought; and whether individual or social preferences define the optimum [8].

Today, psychiatric hospitals are, for example, often required to report their performance on standardised core measures, and to conduct both internal measurement-based as well as external quality improvement activities. Public and private health care providers, consumers and accreditors may use the results several ways to encourage hospitals to improve their quality. In many countries, hospitals are provided with feedback systems comparing their performance with peer organisations, disclosing results publicly in an effort to influence purchaser or consumer decisions, or linking financial incentives to improved performance [14]. Their criteria, however, may vary according to the domains of care regarded as important, the available data sources, and the basis of comparison for the determination of quality.

To increase comparability, the members of the OECD mental health care panel, consisting of international experts in mental health care, suggested a quality indicator set covering the most relevant domains of mental health care (primary care) by selecting four key aspects (http://​www.​oecd.​org/​els/​health/​technicalpapers): treatment, continuity of care, coordination of care, and patient outcomes. Based on a framework outlined by Hermann et al. [20, 21], the panel decided that the indicators should meet the following screening criteria: the indicator measures the technical quality provided, not interpersonal or consumer perspectives; the indicator is focused on quality of care, not on cost or health care utilisation; the indicator is built on a single item, not on a multi-item scale; and the indicator is likely to be useful in quality assessment at the health care system level, rather than the provider level. In addition, Hermann et al. argued that an indicator should likely be constructed from administrative data using uniform coding systems (e.g., ICD or DSM codes), rather than requiring dedicated data collection or non-standardised data elements. The importance of a selected indicator was assumed to be made up of three different dimensions: the impact on health addressing the areas in which there is a clear gap between the actual and potential levels of health, the policy importance, and the susceptibility to be influenced by the health care system. For the latter, the question has to be addressed whether changes in the indicator will give information about the likely success or failure of policy changes.

The scientific soundness of the indicators can be broken down into two dimensions: face validity and content validity [20]. The face validity indicates if the measure is meaningful in a logical and clinical sense. The face validity of an indicator should be based on its basic clinical rationale, and on its past usage in national or other quality reporting activities. The content validity of an indicator addresses if the measure captures meaningful aspects of the quality of care. In addition, the feasibility of an indicator reflects the question of data availability and the burden of reporting [20]. This last dimension should in particular address the issue whether the value of the information contained in an indicator outweighs the costs of data collection and reporting. For mental healthcare, some of the measure attributes conflict with each other. Indicators with the best measurement properties such as utilisation data may not represent the breadth and diversity of the mental health care system in terms of processes, modalities, settings and interventions. For example, whereas psychopharmacological interventions have the broadest evidence base, other areas based on less rigorous studies may be similarly relevant for the long-term course of psychiatric disorders.

Specifications of quality indicators often require the use of approximations, and some measures are a proxy for a broader concept. For example, using hospital re-admission rates as a proxy for the quality of discharge planning assumes that hospital admissions are an unintended outcome. This builds on the ethics of a mental health care system that offers the least restrictive care which is effective. However, there may be research studies showing significant relations between re-admission rates and other measures of quality [1, 45]. Targeted short-term hospital readmissions may even be an indicator of good quality avoiding long-term hospitalisations. Thus, readmissions as quality indicators need to be evaluated in a more-differentiated manner. One of the indicators for provider performance selected by The Mental Health Panel for primary care is the timely ambulatory follow-up after mental health hospitalisation, because the continuity of care was seen as an important aspect of quality in mental health [20]. This indicator can be measured by the number of persons hospitalised due to a primary mental health diagnosis with an ambulatory mental health encounter or with a mental health practitioner within (a) 7 days, or (b) 30 days of hospital discharge. The importance of the indicator is outlined by the fact that most patients with a psychiatric disorder treated in an inpatient setting require follow-up ambulatory care to promote further recovery and prevent relapse. Scheduling outpatient appointments proximally to discharge is generally recommended to address side effects that can result from inpatient medication changes, and to support compliance with the treatment plan. Data indicate that there is wide variability in the duration between hospital discharge and the first ambulatory follow-up visit, some of which is related to patient factors (e.g., severity of illness) and some to system factors (e.g., availability of outpatient appointments). Shorter gaps between discharge and aftercare may contribute to greater continuity of care and a decreased risk of relapse, although research evidence on this question is mixed [15]. More continuity of care may be achieved by improvements in discharge planning interventions that have been shown to be effective in reducing rehospitalisation and in improving adherence to aftercare [39], making this indicator potentially useful in assessing the quality of an integrated care delivery system [37].

A further quality indicator is the hospital readmission rate for psychiatric patients measured by the quota of the total number of readmissions to psychiatric inpatient care that occurred within (a) 7 days or (b) 30 days, divided by the total number of discharges from psychiatric inpatient care during a 12-month reporting period. Hospital readmission rates are widely used as proxies for relapse or complications following an inpatient stay for psychiatric and substance use disorders [20]. Since they indicate premature discharge or lack of coordination with outpatient care, high readmission rates have led some inpatient facilities to examine factors associated with readmissions, including patient characteristics, length of stay, discharge planning, and links with outpatient care [27]. Given the high cost of institutional care, reducing readmission rates can have a substantial effect on mental health spending. However, the relation between readmission rates and other quality criteria are far from consistent. One study showed no association between readmission rates and clinical measures of treatment quality [27] suggesting that it is not the success of the hospital intervention per se which influences the likelihood of readmission. This measure is therefore not a useful measure for one institution but rather for a whole mental health care delivery system, and indicates the quality of post-discharge outpatient treatment as well as the quality of inpatient treatment and information management between mental healthcare sectors.

A further process measure proposed for provider performance measurement is the rate of persons with a mental illness receiving continuous medication treatment in the maintenance phase for disorders like major depression. In this case, the indicator may be specified as the number of persons treated with anti-depressant medication for a period of at least 180 days divided by the number of persons who are diagnosed with a new episode of depression. Depressive disorders can impair personal, social and family functioning, decrease work productivity, and increase the risk of suicide. They are highly prevalent and very disabling. The World Health Organisation has estimated that by the year 2020, major depression will be the second leading disorder in terms of the global burden of disease [33]. Studies have consistently demonstrated that compared with their non-depressed counterparts, individuals with depression experienced impaired physical and role functioning, more workdays lost, and decreased productivity. Many studies show heavy utilisation of health services, with hospitalisations accounting for a high proportion of costs. A range of antidepressant medications have been shown to be effective in ameliorating symptoms, and in improving quality of life and social functioning. However, adherence remains a problem when the choice for a medication trial has been taken with patients who discontinue their antidepressant early having a high risk to experience relapse or recurrence [30]. Randomised clinical trials have provided evidence that antidepressants need to be continued for 4–9 months after initiation to minimise the likelihood of relapse. The health system has considerable influence on this indicator of medication adherence with clinicians playing an important role in influencing patient attitudes to treatment by providing education, addressing concerns, and evaluating and treating side effects [6, 36]. The indicator has good face and content validity. It assesses the effectiveness of clinical management in achieving medication adherence as the basis of the effectiveness of an established dosage regimen by determining the percentage of adults who complete a period of continuation phase treatment adequate for defining a recovery according to the US Agency for Health Care Research and Quality (AHRQ) criteria. Adherence indicators can be constructed from pharmacy data, which may be easily useful to identify patients who need assistance with medication adherence [43]. At the same time, there is no consistent evidence that adherence to antidepressant medication dosages and other guideline recommendations are sufficient to improve patient outcomes. In one study, the authors found no differences in mean endpoint depression scores between a depression guideline intervention group and the control group, so that depression scores were only marginally better in the intervention group [44]. Other studies, however, showed considerable advantages of guideline-based intervention focusing on structured medication treatment and adherence compared to a group with treatment as usual in self-reported and physician-assessed depression scores after 12 months [41].

A possible outcome indicator on the population level in primary care could be the mortality of persons with severe psychiatric disorders specified as the standardised mortality rate for persons with particular psychiatric disorders. Individuals with schizophrenia and other severe mental illnesses have higher age- and sex-adjusted mortality rates than members of the general population [35]. Studies in some countries have found medical conditions and co-morbidities to be under-detected and under-treated in individuals with psychiatric disorders [11]. Such relative mortality rates, which are frequently used in cancer epidemiology studies, are well-accepted and plausible measures to indicate and evaluate the excess mortality in subgroups with certain diseases. They may also provide an estimate of the impact on longevity of these diseases. As there is no a priori biological reason why patients with mental health disorders should die prematurely, a large survival difference between different regions and different mental health care systems could point to shortfalls in the overall quality of medical care, not just mental health care, for this especially vulnerable group of patients. This may also provide a starting point for further investigation.

Aggregate-level measures such as mortality or suicide rates are reliant on external and aggregate sources of data for interpreting results. Inferences can only be made for a whole population and not for the individual patient. When there is no intrinsic standard, a process or outcome measure may be used to identify problems or outliers in the performance of a mental health care system, although such a measure cannot determine the appropriateness of single interventions. Measure performance is heavily influenced by patient characteristics, the case-mix. For example, when hospitals are to be compared, statistical case-mix adjustment is needed to remove the influence of patient characteristics on the results in order to avoid unfair comparisons [23]. In other areas, adjustment of process measures is not required such as suicidality assessment in depression.

Quality measurement is limited by the availability of data. To assess patient outcome, documentation systems for primary care as well as documentation systems for secondary mental health care may be used. This assessment should include instruments to record the first consultation/admission, weekly consultations, and discharge [18]. For specialised care, these could assess sociodemographic data, diagnosis, disease-specific history, treatment course, and outcome. For major depressive disorder, such outcome assessments have been suggested to include the Clinical Global Impressions (CGI) and the Global Assessment of Functioning Scale (GAF), the rating of depression scores by the self-rating Patient Health Questionnaire (brief PHQ-D), and the Beck Depression Inventory (BDI) as well as expert ratings using, i.e., the Hamilton Depression Scale (HAM-D). General aspects of patients’ satisfaction could be measured by the Client Satisfaction Questionnaire [18]. However, a major problem has been identified in routine assessment of patient outcomes. A therapist who is responsible for delivery of a clinical intervention, and thus has a stake in the outcome of the intervention, is not in a position to objectively assess the outcome. Incentives will probably affect rating behaviour most strongly when the individual provider’s job performance is directly assessed [4]. Thus, a rationale approach for quality measurement may combine the assessment of “hard” outcome parameters such as mortality, job integration, and long-term disease course including relapse and remission rates combined with an evaluation of process measures and treatment adherence. Data suggest that the assessment of treatment adherence might have a more positive effect on clinical practice than routine outcome measurements [13].

A cautionary note appears also necessary for payment for performance schemes in psychiatry using indicator systems. Research indicates that clinicians are influenced by the implications of a new quality and outcomes framework indicator when recording diagnoses, disease severity, or treatment processes [9]. Non-incentivised activities and patients’ concerns may receive less clinical attention [28]. Thus, pay-for-performance or prospective payment schemes based on quality indicator performance may hold their promise to improve outcome quality only when such measures are valid and feasible, adopt a longitudinal perspective on quality management, and are successful at ensuring that all providers who are responsible for a particular patient’s care are held accountable for the quality of care they provide [5].

Quality management measures are necessary to analyse weak points in routine care, detect opportunities for improvement of care, and check the implementation of guidelines. Because some studies show that adherence to evidence-based guidelines in psychiatry can improve outcomes [32], whereas others have failed to do so [3], we further discuss this point in the following section.

Throughout the 1990s, educational initiatives for implementing guidelines were begun, and evidence-based guidelines have been formulated in many countries. Several countries have initiated national clinical guideline programmes [16, 18, 36]. There is an ongoing debate whether evidence-based guidelines improve patient outcomes. Clinical guidelines aim to improve quality of care by advocating best-practice models and reducing treatment variation [22]. Some authors have even claimed that the term evidence-based guideline should only be used if a positive impact of guideline implementation on patient outcomes following guideline implementation has been shown [26, 42]. However, there is little evidence that guideline dissemination alone affects the behaviour of mental health clinicians or general practitioners. Guideline implementation programmes using complicated and multifaceted procedures or participatory approaches appear to have an impact on professional behaviour [7].

In a systematic review of psychiatric guideline implementation studies, the observed effects on provider performance or patient outcome after implementation were moderate and temporary in most cases. The studies with positive outcomes used complex multifaceted interventions or specific psychological methods to implement guidelines [42]. Interventions associated with better provider performance were multifaceted interventions with ongoing expert consultation, ongoing supervision, or interventions using marketing techniques and psychological theories to overcome guideline implementation obstacles. Only one study using an intensive and costly intervention strategy showed a consistent positive and substantial improvement in the self- and physician-rated psychopathology of depression [41]. Most studies were statistically underpowered to show small intervention effects.

There is, however, some evidence from observational studies that guideline adherence is associated with better outcomes. In a German multisite hospital study, treatment processes and patient outcomes were compared across seven psychiatric hospitals using case-mix adjustment models [23]. Patient structure and treatment processes showed a great variability between hospitals with mental state, chronicity of the disease, and other patient factors being the strongest predictors of clinical outcome. Benchmarking hospitals, a poorer average clinical outcome was associated with lower guideline conformity in a variety of treatment domains, although it is not clear whether better guideline adherence in the hospitals with the best results was a causal factor for their enhanced performance.

Many quality indicators are derived from clinical practice guidelines. Evidence-based guidelines incorporate research evidence and clinical consensus. They provide a useful foundation for quality improvement. The paucity of studies showing positive effects of guideline implementation on patient outcome in mental health care should not discourage quality improvement initiatives based on guideline recommendations. It should be noted that programmes, which are directed only towards increasing guideline adherence, are too simplistic. Measurement-based quality improvement needs organisational changes and may only be successful when positive incentives exist to continually improve treatment quality.