Assessing Outcomes for Consumers in New York's Assisted Outpatient Treatment Program

Involuntary outpatient commitment, also known as assisted outpatient treatment (AOT), is a civil legal procedure whereby a judge can order a person with mental illness who meets certain criteria to follow a court-ordered treatment plan in the community. Involuntary outpatient commitment is a contested policy. Proponents assert that it provides access to needed treatment for people with severe mental illness; opponents contend that the practice is coercive and unwarranted ( 1 , 2 , 3 ). Effectiveness research should help clarify the benefits and drawbacks of the practice, but the results of existing studies are mixed and are also contested.

The study reported here further informs involuntary outpatient commitment policy—with new findings on hospitalization outcomes, utilization of case management services, and medication possession—from a legislatively sponsored evaluation of New York's "Kendra's Law," the largest and best-funded state program of its kind ever to be implemented ( 4 ).

Forty-four states currently have involuntary outpatient commitment laws. Several states, including New York, California, Florida, Michigan, New Jersey, and Maine, have enacted such statutes over the past decade ( 1 ). Use of involuntary outpatient commitment varies widely across and within states. (An article in this special section by Robbins and colleagues [5] describes these differences.). Effective implementation of involuntary outpatient commitment programs—and the question of whether they actually work for consumers and provide a public benefit—remains a concern to policy makers and stakeholders. The tragic shooting deaths at Virginia Tech at the hands of an individual with mental illness for whom outpatient commitment was ordered but never implemented is a grim reminder that passing a law alone does not necessarily mean that needed services are provided to persons with severe mental illness in the community, and there is considerable variability within and across states in how such statutes are implemented ( 6 , 7 , 8 ).

Under outpatient commitment, a judge may order a person with mental illness to adhere to recommended outpatient treatment, but statutes stop short of permitting forced medication. Under these laws, nonadherence with a treatment plan may lead to a request that law officers transport the consumer to a treatment facility for encouragement to comply with treatment or evaluation for inpatient commitment ( 7 ). This relatively weak enforcement authority has led some to argue that the law "has no teeth"; on the other hand, many mental health consumer advocacy groups vehemently oppose these statutes as intrusive and counterproductive ( 7 ). Recent legislation on involuntary outpatient commitment reflects attempts by advocates and opponents to draft laws with greater or more limited statutory authority ( 9 , 10 , 11 , 12 ).

Randomized controlled trials of the effectiveness of involuntary outpatient commitment in North Carolina ( 7 , 13 , 14 , 15 , 16 , 17 , 18 ) and in New York City ( 19 ) reached contrary conclusions, and the positive outcomes reported for the North Carolina study have been criticized on methodological grounds ( 19 ). Noncontrolled evaluations of the effects of these statutes have generally concluded that they may improve treatment outcomes and decrease hospital readmission rates and lengths of stay ( 7 ). The New York State Office of Mental Health (OMH) reported on the largest noncontrolled study of AOT after the initial five years of experience with more than 3,000 patients under Kendra's Law ( 20 ). The state evaluation reported that treatment adherence more than doubled among consumers who received AOT and that these individuals experienced far fewer hospitalizations (77% fewer), less homelessness (74%), and fewer arrests (83%) and incarcerations (87%) than they did in the three years before initiation of their court-ordered treatment ( 20 ).

The study reported here revisited the question of the effectiveness of AOT by analyzing services utilization data collected for a large sample between 1999 and 2007. Data included paid Medicaid claims and case manager reports of inpatient and outpatient services as well as ratings of treatment engagement. We made two kinds of comparisons. Compared with their own pre-AOT experience, to what extent were consumers in the AOT program more likely to receive intensive case management services and psychotropic medications and to what extent were they less likely to be hospitalized and more likely to spend fewer days in the hospital? Compared with consumers who were participating in assertive community treatment (ACT) but not in the AOT program, to what extent did consumers participating in both the AOT program and ACT or intensive case management demonstrate higher levels of engagement in outpatient services and experience decreased rates of hospitalization?

Methods

Sample

The AOT evaluation project ( 4 ) used multiple sources of data to address the main study objectives, each of which are described in the articles in this special section ( 5 , 21 , 22 , 23 , 24 ). We used two main sources of data in this analysis: Medicaid claims and case manager reports. This project was approved by the institutional review boards of Duke University, Policy Research Associates, New York State OMH, and Biomedical Research Alliance of New York.

Medicaid claims and state hospitalization records

Dependent variables. Service encounter data were obtained for 3,609 individuals who received court-ordered outpatient treatment under the program and who were enrolled in Medicaid at any time between January 1, 1999, and March 14, 2007. Medicaid mental health services for consumers with severe mental illnesses are largely provided on a fee-for-service basis in New York. During this period, a total of 5,634 individuals (unique cases) in the state were given AOT orders; the sample analyzed for this study represents the 64% of these individuals (N=3,609) who were also enrolled in Medicaid. Once enrolled, on average they remained enrolled for 81% of the months that they were on AOT. Data were incomplete for 33 persons, which left a sample of 3,576 AOT recipients.

Hospitalizations and the number of days hospitalized per month were measured by using dates of admission for inpatient stays paid by Medicaid. Additional hospitalizations and the associated lengths of stay were identified in a database of admissions to state psychiatric hospitals that were not covered by Medicaid. Most state hospital admissions in New York are upon referral from a local inpatient psychiatry unit. The state hospital admissions data and the Medicaid inpatient service claims were merged into a single database of psychiatric hospitalizations for consumers before, during, and after their receipt of AOT.

Paid claims for psychotropic medications were used to create the medication possession ratio (MPR), a validated proxy indicator of pharmacotherapeutic continuity and adherence ( 25 ). By using the dates that prescriptions were filled, it was possible to calculate the proportion of days in each month that the consumer would have possessed a supply of medication. The MPR was calculated only for medications that were indicated for treatment of the patient's primary psychiatric disorder as diagnosed by a psychiatrist during an inpatient stay, outpatient treatment visit, or emergency department visit. For example, for patients diagnosed as having schizophrenia, the MPR was calculated on the basis of prescriptions for an antipsychotic medication, and prescriptions for other medications not indicated for schizophrenia were excluded. The MPR was dichotomized at 80% or greater, on the basis of findings of prior studies that this level is significantly associated with decreased hospitalization of persons with severe mental illness ( 25 , 26 , 27 ).

We grouped utilization of case management services to allow several sets of comparisons. In the Medicaid claims analyses, ACT and intensive case management were first grouped together in a single measure of receiving any enhanced or intensive services. A general measure of receipt of any case management was also examined, which included ACT, intensive case management, and "blended" and "supportive" case management. For the analysis of case manager-reported services utilization, ACT and intensive case management were examined as stand-alone service modalities and in combination with AOT.

Independent variables. AOT status—whether a consumer was under a court order in any given month—was obtained from an administrative database that tracked the initiation, expiration, and renewal dates of all court orders. Observations were then grouped by duration of AOT, creating four categories for comparison: months preceding AOT, initial six months of AOT, seven or more months of AOT, and no longer under an order.

All multivariable models controlled for region (New York City, Long Island, Hudson River, Central, and Western New York), race-ethnicity (Hispanic ethnicity, black or African-American race, Caucasian race, Asian race, and other), sex, age (at or above the median of 43 years), Medicare eligibility, and diagnosis.

Propensity scores. We used propensity score weighting to adjust the data for possible selection bias associated with nonrandom assignment to AOT and renewal of the court orders. The probabilities of initiation of court orders and renewal were estimated in two multivariable models. The predicted probabilities from these models were used to create propensity scores. Weighting the sample by the inverse of these propensity scores balanced the distribution of baseline predictors of initiation and renewal across sample observations preceding AOT, during the initial AOT period, and under renewed AOT, thus allowing a simulated random comparison between these groups of observations. Our propensity regression models included all available demographic and clinical variables, MPR, and hospitalization history.

Analysis approach for claims data. We used logistic regression for repeated measures to model the probability of binary outcomes; the analysis controlled for time, relevant covariates, and multiple observations for each individual; our unit of analysis was the person-month. Length of stay—a count outcome—was evaluated with a negative binomial regression model because of overdispersion of the data. All analyses were conducted with SAS, version 9.1.

Case manager reports

Through the Child and Adult Integrated Reporting System, case managers in New York complete a standardized assessment for all AOT and ACT consumers at the initiation of services and every six months thereafter. With these data, we assessed the likelihood of inpatient admissions and levels of case manager-reported engagement in treatment for all individuals who received at least 12 months of ACT treatment alone (N=952), ACT plus AOT (N=852), or intensive case management plus AOT (N=1,734) between January 1, 1999, and July 30, 2007. (Note that this is a different sample than employed for the Medicaid claims analysis described above.) Baseline information for each individual was carried forward for every six-month person-period. Our analyses included person-periods beginning with case managers' retrospective reports at the 12-month follow-up visit—that is, reports covering the period from six to 12 months and including every six months thereafter.

Dependent variables. Case managers reported whether individuals had experienced psychiatric hospitalization in the prior six months and rated the individuals' motivation to engage in services every six months. Motivation to engage in services was defined by case managers' responses to the 5-point scale question: "Indicate which option best characterizes consumer's engagement in services." Choices ranged from 1, not engaged, no contact with provider(s), does not participate in services at all; to 5, excellent, independently and appropriately uses services. Scores of 4 or 5 were considered to reflect engagement in treatment.

Independent variables. The treatment status of consumers was reported by case managers at six-month intervals. These services could include receipt of ACT without a court order for AOT, receipt of ACT with an AOT order, and receipt of intensive case management with an AOT order. Baseline engagement was determined from the initial case manager report for each individual and was included with demographic and related data in models as a covariate. Substance use, Global Assessment of Functioning score, and medication adherence were also included in the models, but unlike the single baseline score for engagement these varied at each six-month report. Substance use was defined as positive if the case manager reported that the consumer had used any alcohol or illicit drugs within the past month.

Analysis approach for case manager data. As with the claims data analysis, we used logistic regression for repeated measures to model the probability of binary outcomes; the analysis controlled for time, relevant covariates, and multiple observations for each individual. Our unit of analysis was the person-month. To adjust for missing data, we used multiple imputation techniques. All variables had ≤5% missing data, and sensitivity analyses that omitted records containing missing data did not substantially change the results we obtained. We calculated odds ratios and predicted probabilities from our resulting logistic regression models for each dependent variable.

Results

Characteristics of the Medicaid sample

As shown in Table 1 , approximately three-quarters of AOT consumers were located in the New York City region of New York State (76%). The second largest proportion was in Long Island (12%). The remaining three regions of the state had substantially fewer AOT consumers. (For a discussion of these regional differences, see the article by Robbins and colleagues [5] in this special section.) The average age of AOT consumers was 42 years, and approximately two-thirds (68%) were male. Forty-two percent of AOT consumers were African American, and just under one-third (31%) were white. Twenty percent of AOT consumers were Hispanic. Approximately four-fifths of AOT consumers had a diagnosis of schizophrenia (82%); the next most prevalent diagnoses were bipolar disorder (12%), major depressive disorder (4%), and other diagnoses (2%).

Enlarge table

Table 2 summarizes findings from the multivariable analysis of associations between duration of AOT (one to six months and seven months or more) and the main outcome variables—hospitalization and days hospitalized, medication possession, and receipt of ACT or intensive case management. [More detailed findings from this analysis, by demographic and clinical characteristics, are presented in an online supplement to this article at ps.psychiatryonline.org .]

Enlarge table

Outcome analyses for Medicaid and OMH claims data

AOT consumers' likelihood of psychiatric hospitalization was significantly lower compared with their pre-AOT experience. During the first six months of an initial order, hospital admissions were reduced by roughly 25% from the pre-AOT period (OR=.77) ( Table 2 ). During any six months of a renewal, the likelihood of an inpatient admission was reduced even further (OR=.59) compared with the pre-AOT period.

To test whether AOT also reduced the average duration of hospital stays, we calculated the average number of days hospitalized during months in which a hospitalization occurred and used this average to compare pre-AOT, short-term AOT, and longer-term AOT periods. (We excluded from this analysis the initial hospitalization when a court order was initiated.) During the pre-AOT period, in months with any hospitalization, the mean±SD number of days hospitalized per month was 18.0±1.5. This average was reduced to 11.0±1.1 days per hospitalized month during months one through six of court-ordered treatment, and 10.0±1.1 days per hospitalized month during continuing months under AOT. Reductions in the number of days hospitalized during any given month were evident in the initial court order and in subsequent renewals of an order (OR=.80 and .84, respectively) ( Table 2 ).

The likelihood of possessing medication for 80% or more of the days in any month was higher during the AOT period. Specifically, the likelihood increased by nearly 50% from the pre-AOT period to the initial six-month period of court-ordered treatment (OR=1.47) and by nearly 90% during the subsequent renewal of the six-month court order (OR=1.88) ( Table 2 ). Similarly, compared with the likelihood in the pre-AOT period, the likelihood of receiving ACT services increased roughly fourfold for both the initial six months (OR=4.13) and the subsequent six-month renewal of the court order (OR=4.03). Increases were also noted in the likelihood of receiving either ACT or intensive case management in any given month of an initial order or a renewal (OR=2.42 and 2.82, respectively) and in the likelihood of receiving any form of case management services during an initial order or renewal (OR=2.65 and 3.48, respectively).

Outcomes for inpatient admissions, length of stay, and medication possession were also examined separately for each region and were essentially unchanged (data available from the authors upon request).

Outcome analyses for case manager data

Analyses of case manager data from the Child and Adult Integrated Reporting System provided an opportunity to compare outcomes for ACT consumers who were not subject to AOT with those of consumers who received ACT or intensive case management under an AOT court order. (A comparison of the demographic characteristics of these groups is available from the authors upon request.) As seen in Table 3 , compared with receipt of ACT alone (that is, ACT without AOT), the addition of a court order to receive ACT significantly reduced the likelihood of hospitalization (OR=.43) during any six-month period after the first six months of AOT. The combination of AOT with other forms of intensive case management was associated with a 43% reduction in the likelihood of hospitalization (OR=.57) compared with receipt of ACT alone. Compared with receipt of ACT alone, receipt of AOT combined with ACT or AOT combined with intensive case management significantly increased the likelihood of high engagement in services as rated by case managers (OR=2.13 and 1.98, respectively).

Enlarge table

Discussion

This study used data from New York's AOT program to examine whether consumers made meaningful gains during outpatient commitment. We used two sources of data to examine key consumer outcomes. In the first approach, based on Medicaid and New York State OMH claims data, outcomes for consumers were compared with their own experiences before receiving AOT. Compared with the pre-AOT period, consumers under a court order were significantly less likely to be hospitalized for psychiatric treatment, spent fewer days when hospitalized, were more likely to possess an adequate supply of psychotropic medication appropriate to their diagnosis, and were more likely to receive ACT, intensive case management, or other forms of case management in the community—all important and policy-relevant outcomes.

However, this analytic approach was limited by its lack of a contemporaneous comparison group. To address this limitation, we used another approach, building a natural comparison group from data available in the state's case manager reporting system. All persons who receive ACT or AOT are evaluated every six months by case managers. Because most AOT consumers are also served by ACT or intensive case management, the data allowed for a comparison of outcomes of those who received ACT alone with those who received ACT plus AOT or intensive case management plus AOT. In these more direct comparisons, the court order reduced hospitalizations over and above the effect for ACT alone. These results more directly demonstrate that AOT itself offers benefits in addition to those of ACT—the exemplar of evidence-based community treatment.

There are additional limitations to this study. The analysis did not control for all the reasons that some individuals received longer or shorter periods of AOT or the myriad reasons for ending the court order at a given time. We were able to control for some but not all theoretically relevant covariates. The outcomes selected for evaluation, such as hospitalization, may not be universally accepted as the most relevant or important outcomes for evaluating the effectiveness of involuntary outpatient commitment; some stakeholders would place higher value on other outcomes. In addition, inpatient admissions could show regression to the mean over time, given the high rate of admissions experienced by individuals who eventually receive AOT. However, these analyses in most cases incorporated long periods of observation before a court order, which would include many periods of illness exacerbation and improvement; thus regression to the mean is unlikely.

Reliance on Medicaid claims data for many of these analyses may fail to reflect the experience of consumers not enrolled in the Medicaid program or fail to reflect experiences during periods of ineligibility. In addition, AOT consumers may have been more likely to be consistently enrolled in Medicaid. Data from the case managers offers a partial remedy, in that their reporting is unaffected by Medicaid eligibility, but it may be subject to several other sources of bias. Because New York's AOT program is unique in its comprehensive implementation and infusion of new service dollars into a relatively well-funded mental health system, these findings may not generalize to other states. Ultimately the most stringent and methodologically sound test of the effectiveness of New York's involuntary outpatient commitment program would require a carefully conducted randomized trial.

Conclusions

This study conducted several complementary analyses of the largest data repository available on an involuntary outpatient commitment program. Findings associated with AOT include reductions in psychiatric hospitalizations, improved rates of psychotropic medication possession, enhanced case management services, and improvements in related outcomes.

Acknowledgments and disclosures

The study is funded by the New York State Office of Mental Health, with additional support from the John D. and Catherine T. MacArthur Foundation Research Network on Mandated Community Treatment. The authors gratefully acknowledge the contribution of numerous individuals in collecting, synthesizing, and reporting the data for this effort. At Policy Research Associates, they thank Karli Keator, Wendy Vogel, M.P.A., Roumen Vesselinov, Ph.D., Jody Zabel, Steven Hornsby, L.M.S.W., and Amy Thompson, M.S.W. They also acknowledge the extensive reviews and critical feedback provided by the MacArthur Research Network on Mandated Community Treatment, which served as an internal advisory group to the study. Although the findings of the study are solely the responsibility of the authors, they gratefully acknowledge the support and assistance of the New York State Office of Mental Health in completing the report, including Steve Huz, Ph.D., Chip Felton, M.S.W., Peter Lannon, Susan Shilling, J.D., L.C.S.W., Qingxian Chen, Michael F. Hogan, Ph.D., Bruce E. Feig, and Lloyd I. Sederer, M.D.

The authors report no competing interests.