Performance indicators for public mental healthcare: a systematic international inventory

The library-database and internet search resulted in 3193 publications in English- and Dutch- language peer-reviewed journals and websites from governmental as well as nongovernmental organizations. Further selection based on title- and keyword criteria resulted in the inclusion of approximately 480 publications. After reviewing the abstracts, 152 publications on quality measurement in adult (public) mental health care were included. Final selection based on full publication content resulted in the exclusion of another 46 publications that did not explicitly specify the measures applied to assess health care performance, leaving 106 publications to be included in the final sample.

Table 1 shows the included publications structured by nation/region and date of (first) publication.

Publications on indicator development, implementation, and validation within ten nations were found. Three international organizations (i.e. European Union, OECD, and WHO) developed PI for between nation comparisons. The majority of the publications (n = 90, 85%) focus on the quality of PMHC in nations where English is the native language (Australia, Canada, United Kingdom, and USA), and 66 publications (61%) are concerned with PMHC in the United States. In contrast, publications that focus on the measurement of PMHC quality in Spain, Germany, Italy, South Africa, the Netherlands, and Singapore together only account for 12% of the total sample. The majority of the publications were found in peer-reviewed journals (n = 65; 61%), the remaining publications (n = 41; 39%) consisted of reports, bulletins, and websites by governmental and non-governmental organizations. In the next sections, the performance measurement initiatives and publications per nation/region are discussed.

A total of 1480 unique PI are included in the inventory. 370 indicators of these are represented in two or more publications. To assess individual PI we focused on characteristics reported by the developers in terms of method of development, level of assessment, Donabedian's domain of care, dimensions of performance, diagnosis or condition, and data source, as presented in Table 2.

More than a quarter of the PI are based solely on expert opinion and more than half of the PI are developed using both literature review and expert consultation, with 12% utilizing a structured consensus procedure such as adaptation of the RAND-method or a modified Delphi procedure [e.g. [21, 79, 82]]. For 59 of the included PI no method of development was specified in the publication.

With regard to the level of assessment, the majority of the PI (55.7%) aim to assess the quality of a PMHC system. These PI incorporate data from multiple service providers within a region (e.g. county, state, province, nation) to measure the standard of PMHC-quality either against benchmarks set by the regional legislator, or against the PMHC-quality in other regions. Only 7 PI were specifically designed to assess the performance of individual clinicians, i.e. physicians in a community mental health center [24]. The remaining PI (43.9%) measure the performance on a service-level to improve quality of care, gain transparency for purchasers, or inform patient-choice.

We found more than a third of the PI measure performance in terms of treatment outcome, for instance in suicide rates, crime rates, or incidence rates of homelessness. Almost half of the included PI are process measures. Based on guidelines that specify 'best practice' treatment processes, for instance in terms of duration, contact intensity, or medication dosage, these PI are usually formulated as a proportion of a population that is provided with a treatment according to guidelines. However, there are differences in both the guidelines used (e.g. the number of days between discharge from inpatient treatment and the first outpatient contact varies between 1 and 30), and in the population used in the denominator of the PI (e.g. the percentage of the population in a region vs. the population that receives treatment).

Dimension of performance is the most diverse categorization of PI. To structure the PI from different regions and developers by dimension of performance, some concepts that are strongly related were grouped, resulting in eight main dimensions of performance. 42.8% of the PI aim to measure the 'effectiveness' of PMHC and provide stakeholders with a measure of degree of achieving desirable outcomes. Another 19.5% of the PI are designed to measure the 'accessibility' and 'equity' of PMHC. Remarkably, only 2.9% of the PI assess the efficiency, cost, or expenditure of PMHC.

A large majority of the PI (69.9%) is not specific to any diagnostic group or a group with a certain condition (such as homelessness). Relatively many of the diagnosis-specific PI are developed for psychosis-related disorders (8.4%) and substance abuse disorders (8.2%), as many individuals in the target-group of PMHC cope with these disorders.

Finally, the data source of the included PI was inventoried. For a considerable number of PI (22.5%), no data source was specified. Development of these PI did often include the specification of the data needed for the PI, but did not identify a data source. More than half of the PI for which a data source was specified are based on administrative data or medical records.

Aspects of feasibility and content validity of PI can be established through literature review and expert consultation. Almost all PI development initiatives have used literature review and/or expert-consultation methods to establish at least some evidence on the content validity of the proposed PI. Furthermore, through stakeholder-consultation techniques, ranging from telephone interviews and expert meetings to structured consensus procedures, possibilities for implementation and support for the PI in the field is often assessed as well, thus establishing evidence for the feasibility of the PI. With regard to content validity and feasibility (i.e. support in the field and expected reliability) we confine ourselves to remarking that for all the PI included in this inventory, some evidence for the content validity and feasibility has been established. However, the strength of the evidence varies and depends heavily on the methods used to consult stakeholders and experts [126].

Although forms of reliability that are relevant to PI based on surveys and audits, such as test-retest reliability and inter-rater reliability, have been examined by several indicator developers [e.g. [71, 76, 88]], reliability in terms of accuracy and completeness of the (administrative) data sources used for PI has rarely been assessed. A number of organizations and authors have recognized that information contained in databases and patient records may be incomplete, inaccurate, or misleading which can have a considerable effect on the usability and feasibility of the PI [e.g. [21, 52, 69]]. However, we found only two studies that assessed the accuracy or availability of the information needed for the PI. Huff used data from the Massachusetts state contractor's claims data set used for paying providers and data from the Medicaid beneficiary eligibility file. The author considered key fields in the claims dataset to be of high reliability because those data elements are essential in determining service reimbursement and the timeline of the payments. The eligibility file had some reliability problems with a small subset of duplicate cases and missing values in specific fields, in particular 'beneficiary race'. Huff concluded that both data-files were sufficiently reliable, at least for the intended study [32]. Garcia Armesto et al. provided an overview of mental health care information systems in 18 OECD countries to support the implementation of the OECD system level PI selected in 2004. They conducted a survey in each of the participating countries to gather information on the types of system-level mental health data available, the data sources available on a national level, and the institutional arrangements on ownership and use of the information systems and concluded that data on mental health care structures and activities is generally available but data necessary for measurement of mental health processes and outcomes is more problematic. Furthermore, the integration of information systems across different levels of care provision (i.e. inpatient, outpatient, ambulatory, and community care) was found to be low [119]. The mixed results presented by these two studies show that data reliability (accuracy as well as completeness) cannot be presumed to be sufficient for the implementation of PI for PMHC.

We found 18 publications that focused on the relation between an indicator and an external criterion. A broad range of criteria have been used in these assessments of the criterion and construct validity of PI. The criteria vary in perspective (covering subjective quality or technical quality), domain of care (measures of structure, process or outcome), and in data source (questionnaires, audits, or administrative data). The studied PI, the criteria used to validate the PI, and the outcome of the study are shown in Table 3.

Six studies have focused on the relations between measures of (client) satisfaction and indicators of technical PMHC quality. Either by assessing the relation of a measure of satisfaction with an external criterion [17, 22] or using measures of satisfaction as criteria for PMHC quality to study the usefulness of PI of PMHC processes and (clinical) outcomes [30, 59, 91, 110]. Four of these studies show significant associations between the satisfaction measure with measures of effectiveness, appropriateness, accessibility, and responsiveness. Another study reported relations between measures of satisfaction and measures of effectiveness and appropriateness as well, but those associations disappeared when client-level data were aggregated to reflect the quality on a service level of assessment. A relation between satisfaction and a measure of continuity of care, however, did remain significant on both levels of assessment. One study did not find any association of measures of client satisfaction with community-valued outcome indicators, such as involvement in meaningful activities or residential independence. Thus, we found evidence to support the criterion validity of satisfaction measures, specifically as measures of the continuity of PMHC. However, measures of client satisfaction seem to be less useful in the assessment of the long-term effect of PMHC on a population in its catchment area.

Two studies assessed the validity of preadmission care as an indicator for the quality of public mental health services [44, 111]. Both studies found a relation between preadmission care and post-discharge use of care. The associations of preadmission care measures and measures of readmission found in these studies were more mixed. Wierdsma et al. found no relations between preadmission care and readmission within 90 days or within one year, and Dausey et al. found that clients who received preadmission care were slightly more likely to be readmitted within 14, 30 and 180 days after discharge. The studies show contradictory results on the associations of preadmission care and the length of stay. In one study length of stay is increased when the clients receives any care before admission, while the other study reports a decrease in length of stay when any preadmission care is received. Based on the results of these studies, receiving preadmission care can be considered to be useful in assessment of the continuity of PMHC. However, the validity of 'length of stay' as a criterion for PMHC quality is questionable. A study by Huff used readmission within 30 days after discharge for an acute mental health care need as a criterion measure and showed no association of median length of stay and this criterion [32]. Length of stay was the only measure of the eight measures assessed in this study that showed no relation with the criterion.

PI for the appropriateness of depression care, and their relation to mental health care outcomes and structures were assessed in three studies [62, 64, 70]. The PI assess appropriateness expressed as the ratio of clients receiving outpatient depression care, which receive guideline-conformant medication dosage, medication duration, and follow-up visits. The results of these studies with regard to the criterion validity of these PI vary. The two studies that assessed the relation of dosage adequacy did not find an association with the outcome criteria. Two studies that assessed (measures of) appropriate medication duration did find associations with volume of care, and post-care period hospitalizations, but the one study that included medication duration in an indicator of appropriate medication did not find a relation with the outcome-criterion absenteeism. The results on measures of follow-up visit adequacy were mixed as well. Associations with volume and absenteeism were shown, but no relation to post-care period hospitalizations was found.

Two studies proposed PI that included aspects of service need and assessed their usefulness [41, 66]. In showing that these PI are associated with PMHC processes (i.e. average caseload and provided service hours) and outcomes (i.e. medico-legal and suicide mortality rates), these studies contributed to the evidence on the usefulness of PI that incorporate service-need of clients.

The validity of PI developed by (semi-) governmental organizations was assessed in only two studies published in peer-reviewed journals. Druss et al. used HEDIS measures of medication management and follow-up to assess the volume-quality relationship and found significant associations between both follow-up and medication management measures, and volume of mental health services [64]. A study by Beaulieu et al. assessed the criterion validity of the performance measurement instrument for local public health systems, developed by the CDC [50]. They reported on the association of responses on the instrument and documentary evidence and found that it validated the responses. However, a second method employed to validate the response on the instrument against ratings by external judges proved to be unreliable due to lack of knowledge of the local systems of the judges.

Three studies assessed the validity of measures of mental health care structure as measures of quality. Macias et al. assessed the potential worth of model-specific mental health program certification as a core component of state and regional performance contracting with mental health agencies. Based on an evaluation of International Center for Clubhouse Development prototype certification program, they conclude that a model-based certification program can attain sufficient validity to justify its inclusion in mental health service performance contracting [28]. Davis and Lowell suggested an optimum ratio of state-operated to community-operated psychiatric hospital beds and assessed the relation of (deviation from) this ratio to suicide rate, and cost of mental health care. The results of these studies show that suicide rate is lower in states in which these ratios were close to the theoretical optimum. The relationship of the optimum ratio and cost per capita was less clear cut. A linear relationship was found only when outliers were excluded [42, 43].

Finally, Nieuwenhuijsen et al. assessed the relationship of ten process measures and their summed score with two measures of outcome i.e. time to return to work, and change in level of fatigue. Time to return to work was found to be related to only three of the process measures and the summed score, and no significant relations between change in level of fatigue and any of the process measures were found. Thus, although content validity of ten of the eleven PI was established, the criterion validity of most of the separate PI was not [110].