Predictors of Admission in First-Episode Psychosis: Developing a Risk Adjustment Model for Service Comparisons

There has been increased interest in the course and outcome of first-episode psychosis, which has been spurred on by both an interest in its underlying neurobiology and the potential for improving clinical outcomes ( 1 ). Several clinical trials have been conducted to assess the efficacy of specialized programs for treatment of first-episode psychosis in comparison with treatment as usual ( 2 ). The results have been sufficiently encouraging for some government health ministries to implement such services as routine care as a matter of policy ( 3 ). However, there is a dearth of published data on the specifics of the care provided or outcomes of treatment as usual in real-world settings ( 4 ).

Performance measures for assessing the outcome of first-episode psychosis have been identified ( 5 ), have been found to be feasible for individual program evaluation ( 6 ), and have been used to provide evidence to support establishing standards of care ( 7 ). In this article we focus on hospital admission as the key performance measure because it has been identified as a good proxy outcome in studies of schizophrenia ( 8 ). As such it should be suitable for measuring the outcome of any form of service to groups with a first-episode psychosis whatever the service delivery model, including networks of individual providers, standard community mental health services, or specialized first-episode psychosis services. Three key attributes of performance measures are that they should be meaningful, feasible, and actionable ( 9 ). Hospital admission is a measure that meets these criteria, particularly for serious mental disorders manifesting as first-episode psychosis. Admission for first-episode psychosis is meaningful in that it is associated with negative outcomes such as relapse, violence, attempted suicide, and increased costs ( 10 , 11 ). It is feasible to measure because it is a concrete measure that is reported in most if not all health services. It is actionable because it is related to processes of care, including inpatient and outpatient care ( 12 ). In the outpatient setting preventing admission has been related to medication adherence ( 13 ) and family psychoeducation ( 14 ). These attributes led us to our study goal, which was to develop a risk adjustment model based on a set of evidence-based predictor variables for hospital admission. We planned to evaluate the model's predictive discrimination using the C statistic, and we considered a value of .7 or higher to signify good discrimination ( 15 ).

Risk adjustment is a statistical process for controlling for group differences when comparing nonequivalent groups on outcomes of interest. It can achieve only a partial adjustment and is not equivalent to randomized patient assignment ( 16 ). Risk adjustment has been used to compare mental health services ( 17 ), but it has not been applied to compare outcomes of either treatment as usual or specialized services for first-episode psychosis. In this study we examined the predictors of hospital admission in first-episode psychosis in order to develop a risk adjustment model for comparing outcomes across services with different patient populations and service delivery models.

Methods

We used a retrospective cohort design to assess hospital admission one, two, and three years after patient enrollment into a first-episode psychosis program that serves a population of about one million. The services are provided at no cost to the consumer at the point of service delivery because all of these patients have health insurance. Admissions to local hospitals are recorded in a single database. Hospital admissions are easy to track because the population is relatively isolated and there are few admissions to hospitals in other regions. Unfortunately the exact number of admissions elsewhere is unknown. Potential risk factors for hospital admission were identified by a panel of experts who used the Template for Risk Adjustment and Transfer (TRAIT) ( 18 ). Multiple regression analysis was then used to construct a model in which each of the candidate risk factors was entered as a separate variable ( 19 ).

The development of the risk adjustment model involved three steps: literature review, expert panel review, and model development. The first step in the research was a literature review that examined potential risk factors for hospital admission. The following terms were searched in MEDLINE and PsycINFO databases in the years 1990–2006: schizophrenia and risk factor, mental illness and risk factor, schizophrenia and risk variable, mental illness and risk variable, predictive factor and outcome, predictor, risk adjustment, outcome, comorbidity, prognosis and outcome, and prognostic factor.

The next step in the research was to have an expert panel reduce the resulting set of 17 candidate variables to a parsimonious set of potential risk adjustment variables with the assistance of TRAIT. The expert panel comprised psychiatric epidemiologists and clinical and health services researchers, all with expertise in first-episode psychosis research. A full description of the method is outlined in a document developed for the Center for Mental Health Services of the Substance Abuse and Mental Health Services Administration ( 18 ). Members of the expert panel ranked the level of evidence for individual risk factors according to three categories—published research evidence, clinical experience, or other experience or evidence. The TRAIT was originally developed to contribute to closing the knowledge gap between developers of quality measures and developers of models to adjust results from quality measures for level of risk. TRAIT is a template that guides measure developers in the identification of patient factors that are potentially useful for risk adjustment of an individual quality measure.

The next step involved identification of two cohorts that each represented an incidence cohort of first-episode psychosis patients from the same catchment area of about one million people. The first sample consisted of 297 patients admitted between February 2003 and December 2005, and the validation sample comprised 309 consecutive admissions between January 1997 and December 2000. All patients received similar treatment in a program for individuals experiencing a first episode of psychosis ( 20 ). Participants met DSM-IV ( 21 ) criteria for schizophrenia, schizophreniform disorder, schizoaffective disorder, or other psychotic disorders. At the time of enrollment they had received less than 16 weeks of adequate treatment for their first psychotic episode ( 22 ).

The power of our available sample was considered to be adequate based on the estimate that five to ten events are required to test each risk adjustor ( 23 ). Our planned sample size of 300 (270 at two years) provided an estimated 80 events (admissions), which was considered adequate because we did not plan to test more than 15 predictors.

Exclusion criteria included evidence of an organic central nervous system disorder (such as epilepsy or traumatic brain injury), mental retardation, or being under age 16 or over age 65. The research project was approved by the local conjoint research ethics committee. In accordance with the local Health Information Protection Act and ethics committee guidelines, individual patient consent was not required because the project used deidentified data from administrative and clinical databases.

In the third stage of developing our risk adjustment model, we used multivariate logistic regression to model hospital admission as a function of the 12 potential risk adjustment variables identified by the TRAIT. The model was first used in the development cohort and then externally validated in a second cohort ( 24 ). Model reduction was conducted using methodology consistent with that of Harrell and colleagues ( 15 ). Model assumptions of logistic regression were tested, and necessary modifications were made. For hospital admission, model discrimination (ability to predict differences in the outcome on the basis of the risk factors) was crudely assessed by plotting covariance graphs. Next, the C statistic was calculated ( 25 ). This equals the area under the curve for the receiver operating characteristic (sensitivity versus 1 - specificity) of the predictive model for the observed data; C=.5 indicates no predictive discrimination, whereas C=1.0 implies perfect discrimination. Bootstrap methods were used to construct a 95% confidence interval for C as detailed by Efron ( 26 ). This is to allow comparisons across models and with the validation data. Bootstrap methods allow calculation of confidence intervals for variables such as the C statistic, which do not have known underlying distributions. The data were analyzed with Stata 10.0 ( 27 ).

Results

Selected terms were searched in MEDLINE and PsycINFO for the years 1990–2006 and produced over 45,000 hits: schizophrenia and risk factor, 1,592; mental illness and risk factor, 25,886; schizophrenia and risk variable, 90; mental illness and risk variable, 980; predictive factor and outcome, 17; predictor, 382; risk adjustment, 39; outcome, 3,958; comorbidity, 1,023; prognosis and outcome, 2,451; and prognostic factor, 11. Therefore, the search strategy was refined to include at least one of the following combinations: Positive and Negative Syndrome Scale (PANSS) and predictors; Global Assessment of Functioning (GAF) and predictors; relapse, risk factor, and schizophrenia; rehospitalization; remission and risk factor and schizophrenia; and predictor and outcome. A review of 18 identified studies produced 17 potential risk factors.

The TRAIT analysis identified 12 of these 17 variables as potential predictors of hospitalization. For this study only one diagnostic category was considered: first-episode psychosis. This meant that no specific diagnoses were considered to be predictors of hospitalization. The TRAIT analysis revealed potential variables, including sociodemographic, clinical, and other attributes. The variables and their definitions are presented in the box on this page.

The clinical and sociodemographic features of the development and validation samples are reported in Table 1 . The samples were broadly similar, but there were significant differences between the two samples in enrollment age for the first-episode program (t=2.9, df=604, p=.004) and PANSS negative symptom score at baseline (t=4.0, df=595, p<.001), proportion of Caucasians (z=-3.1, df=1, p=.002), marital status (z=2.0, df=1, p=.05), and major depression.

Enlarge table

The high percentage of missing data (N=98) in the validation database for high school education necessitated removal of the variable from the modeling process. The one-, two-, and three-year hospitalization outcomes were similar in both samples, with no significant differences between cohorts ( Table 2 ).

Enlarge table

The results of the multiple regression analyses are presented in Table 3 . Prior hospitalization was a common significant predictor of admission in all three time periods in both the development and validation samples, and the associated odds ratios were comparable for the two samples. In the validation sample, comorbid substance use disorder was a significant predictor of admissions by years 1 and 2 (OR=2.50, p=.01; OR=1.84, p=.05, respectively), and GAF score was a significant predictor of admissions by years 2 and 3 (OR=.97, p=.03; OR=.97, p=.047, respectively). The initial PANSS score for positive symptoms was a significant predictor of admission by years 2 and 3 in the development sample (OR=1.07, p=.02; OR=1.06, p=.02, respectively). Finally, the C statistics for the model in the development data were .72 or higher for admissions in all periods and ranged from .67 to .72 in the validation data. The 95% bootstrap confidence intervals had lower bounds of .61 or higher and upper bounds of .80 or lower in all cases. We also examined the strength of the model using only prior admission as a variable. This reduced the strength of the risk adjustment model as assessed by a C statistic of .59–.64.

Enlarge table

Discussion

These results represent the first report of a risk adjustment model for a valid, available, and actionable outcome for patients with a first episode of psychosis. The model will require replication, and perhaps modification and external validation with larger samples, but the model serves as a tool that can be used to enhance comparisons of outcome across first-episode psychosis programs, which could lead to changes that improve the quality of care within a program ( 28 ). The sample size met the requirements of the power calculations; however, it was relatively small for the development of risk adjustment models, which are commonly applied to large populations of patients. With some planning for the data for both the outcome measure and the predictor variables, data could be collected on a scale that would allow for application to large samples and enhance comparison of performance across programs. We have used regression techniques that represent the preferred approach, compared with stratification techniques and an approach that has been used for examination of hospitalization for mental illness ( 29 ). Risk adjustment is an important tool that can be used for quality improvement, more specifically for setting standards and norms for service performance.

As mentioned above, a C value of .5 indicates no predictive discrimination, whereas a value of 1.0 implies perfect prediction ( 15 ). With C=.7 representing good discrimination ( 30 ), the logistic regression models for both of our samples performed well (C=.67–.74). Moreover, no confidence intervals for the C statistic had a bound lower than .61. We also examined the impact of using only prior admissions as a predictor and found that C dropped to the range of .55–.64, which demonstrates that inclusion of other predictor variables strengthened the prediction model. The point estimates of the C statistics for each model in the validation sample were less than in the development sample, as would be expected ( 15 ). However, examination of the confidence intervals suggests that in fact these C statistic estimates were not significantly different. Therefore, the model remained robust in the validation sample, which had baseline characteristics different from those in the development sample. The implication is that this risk adjustment model may continue to discriminate when used to compare outcomes of other programs with diverse patient characteristics.

A limitation of the study is that it examined data from only one region serving a population of about one million. Therefore, it is important to test this risk adjustment model in larger and more diverse populations served by a range of types of services. Many individuals with a first episode of psychosis do not receive specialized first-episode psychosis services, but rates of hospitalization were appropriate for evaluating such general mental health services. Larger studies could also examine other predictor variables, such as financial status and health coverage, in populations where these may vary. Another limitation is that we used secondary data and were therefore unable to include some measures identified by the TRAIT as potential risk factors, such as socioeconomic status. Despite this limitation, the model achieved good discrimination as defined by the C statistic. A further limitation lies in the use of hospital admission as a key performance measure for first-episode psychosis services. Much of the data supporting its use as a performance measure were drawn from studies of multiepisode schizophrenia. There is a need for prospective studies in the first-episode population linking processes of care to this key outcome measure.

Conclusions

We have developed a risk adjustment model that corrects for baseline patient characteristics when comparing hospitalization outcomes after initiation of treatment for first-episode psychosis. The model was robust and was validated in a second sample. The next step should be to test the model in larger populations and across a range of services. Such analysis would require programs to collect baseline data on the variables identified in this project as predictors of hospitalization.

Acknowledgments and disclosures

This research was supported by a grant to Dr. Addington from the Alberta Heritage Foundation for Medical Research.

The authors report no competing interests.