Adjusting for geographic variation in observational comparative effectiveness studies: a case study of antipsychotics using state Medicaid data

Tremendous progress in codifying best practices for conducting comparative effectiveness research (CER) using observational datasets has been made. The Agency for Healthcare Research and Quality (AHRQ) [1] and Patient Centered Outcomes Research Institute (PCORI) [2] have published detailed protocols outlining standards for the design and implementation of CER studies. These reports stress that observational CER studies require multidimensional analytic strategies and careful consideration of which multivariate modeling approaches best represent the structure of the data, the study outcome, and exposure/treatment being studied. What is largely absent from the CER literature is an explicit discussion of how geographical context can affect observational studies and, in particular, introduce bias into statistical analyses (see, for example, the August 2013 special supplement on methods for CER/PCOR in the Journal of Clinical Epidemiology). Geographical variation in populations, physician practice patterns, and the availability and accessibility of hospital, specialty, and primary care can influence both treatment choices and care seeking behaviors [3‐6]. These differences may, in turn, be reflected in health-care utilization databases often used in CER studies [7]. Spatial statistical methods, which examine small-area variation in these factors, can provide additional information to assist in further alleviating bias in observational CER studies.

The Dartmouth Atlas of Healthcare series [8‐10] and the Atlas of U.S. Mortality[11] provide strong empirical evidence of geographical patterns of mortality, disease, health care resource availability and service utilization across the U.S. Since their publication in the mid-1990’s, numerous studies have demonstrated geographical differences in health and health care for many outcomes and procedures. This literature suggests that two geographic principles govern spatial patterns of care. First, people with similar demographic characteristics and risk factors tend to live near each other, producing larger areal patterns of morbidity and mortality [12]. This is often referred to as the compositional effect, since the geographic patterns are due to the underling composition of the population. These patterns, for a variety of socio-demographic reasons, often translate to variations in service use, particularly hospitalization and emergency department (ED) visits. For example, a county with more low income individuals may exhibit higher rates of ED visits for asthma because the condition is not being managed well due to lack of primary care. These patterns are particularly important for observational CER studies, which rely on electronic medical records, insurance claims or hospital discharge data to measure outcomes. Many of the analytic strategies suggested by AHRQ and PCORI attempt to account for this variation by controlling for, (e.g., multivariate models) or matching on (e.g., propensity scores), patient-level characteristics. If the area-level differences are indeed due to compositional effects, these techniques should be effective in controlling for bias.

The second geographic principle which drives spatial patterns of morbidity and utilization is the notion of areal contextual differences. The care a person receives depends upon the supply of resources available and physician practice patterns in the place the person lives [13]. Recent CER work has introduced instrumental variables (IVs) to capture geographic variation in practice patterns which may affect treatment exposure [14‐16]. Unfortunately, one condition of an IV is that it: “is a cause of treatment but has no causal association with the outcome other than through its effect on treatment” [1], which is difficult for many CER studies. The social epidemiology and sociology literature also suggests that health outcomes, service utilization and treatment decisions are affected by social and cultural norms, social ties and neighborhood disorganization, often measured using area-level socioeconomic status [17‐21]. The current literature on geographic variation in health has shifted from descriptive analysis of spatial patterns to sophisticated statistical analyses of the contextual factors which contribute to these patterns [22‐26]. Contextual factors are ubiquitous in social epidemiological analyses but rarely considered in CER. IVs cannot be used to control for area-level socio-economic or cultural patterns because these variables are often highly related to the outcome under study. As such, there is a need to integrate valuable knowledge from geography, social epidemiology and sociology into CER best practices in order to explore the effect of geographical variations in area-level socioeconomic status (SES) and utilization on both the treatment exposure and the outcome under study.

In this study, we seek to build upon the strong analytic foundation laid by AHRQ and PCORI by suggesting that exploratory spatial data analysis (ESDA) and spatial statistical methods be integrated into CER studies and included in guidelines for rigorous CER methods. This study is based on two hypotheses related to two potential sources of confounding bias: 1) drug prescription patterns (our study exposure) may vary geographically as adoption may be uneven throughout the state, and 2) hospitalization and ED visits (our study outcomes) may vary geographically depending upon local referral and treatment practices and availability of health services (e.g., in urban vs. rural areas). Therefore, we investigate whether area-level geographic variation in treatment and outcomes is a source of confounding bias in this observational comparative effectiveness study. ESDA is a powerful tool which can be used to understand the potential biases produced by area-level compositional and contextual factors in CER studies, while spatial statistical methods can control for these biases. We demonstrate these techniques, and their utility, with a case study that compares the effectiveness of paliperidone palmitate, a long-acting second-generation antipsychotic (SGA), with orally-administered SGA’s among Medicaid patients in the State of Missouri. If area-level variation is a source of confounding bias, it should be investigated as part of good comparative effectiveness research practices.

This is a secondary analysis of a retrospective observational study using a new user cohort design. Although we have examined these data using other methods appropriate for observational data (e.g., propensity score matching and provider and patient restricted cohorts), we do not present these analyses here. Rather, we focus on additional geographic analyses not typically included in observational CER studies which aim to determine how geographical differences in the treatment and outcome may affect results. First, we conducted ESDA by: 1) mapping drug utilization, and hospitalization/ED visit rates, 2) smoothing those rates using Spatial Empirical Bayes (SEB) methodologies, and 3) analyzing spatial patterns using Moran’s I and local indicators of spatial association to look for local clustering. Second, we developed multilevel logistic regression models and mapped random effects to evaluate model performance and the effect of spatial differences in utilization and admission rates on the interpretation of our results.

This study received approval from the University of Colorado Denver Multiple Institutional Review Board (IRB) and adhered to Data Use Agreements specified by the State of Missouri.

Figures 1 and 2 present results of the exploratory spatial data analyses for drug utilization rates (Figure 1) and hospitalization and ED visit rates (Figure 2). Figure 1A and B show the SEB smoothed rates for paliperidone palmitate and oral SGAs. Results indicate a clear spatial pattern of oral SGA use in the southern part of Missouri, which is predominantly rural with small towns and sparse population. Qualitatively, paliperidone palmitate utilization rates were higher in the southeast corner and north central portion of the state relative to other regions. Utilization of both drugs was higher in the major metropolitan areas of St. Louis and Kansas City. Figure 1C and D show the results of the Moran’s I and LISA analyses. The Moran’s I indicates that there is statistically significant, but weak, clustering of paliperidone palmitate (I = 0.05; p = 0.004) and a stronger clustering effect of oral SGAs (I = 0.16, p < 0.0001). The LISA maps indicate paliperidone palmitate had no strong visual pattern of local clustering (which is also reflected in the low Moran’s I of 0.05), but oral SGAs exhibited clusters of high use in the southeast corner, and a cluster of low utilization to the north of Kansas City and around St. Louis (which is reflected in a higher Moran’s I of 0.16).Figure 2A and B show the SEB smoothed rates for ED visits and hospitalizations. Results indicate no clear spatial pattern for ED visits and hint at slightly higher hospitalizations in the southern portion of the state. The Moran’s I and LISA results strengthen this conclusion in Figure 2C and D. The Moran’s I is not statistically significant for ED visits, though LISA maps indicate the ED visits are high in the city cores of both Kansas City and St. Louis. The Moran’s I for hospitalizations is weak and borderline significant (I = 0.03; p = 0.05), and the LISA map indicates several larger clusters of high hospitalization in the southeast corner, and clusters of low rates in the northern region of the state.

Taken together, these results indicate that treatment exposure varied across regions, qualitatively indicated by visual interpretation of spatial patterns and quantitatively confirmed by statistically significant Moran’s I and LISA values, and that moderate spatial differences in the outcomes used to examine the effectiveness of paliperidone palmitate exist. The maps also qualitatively suggest both county-level and MHSA-level patterns exist for drug utilization.

Additional file 1: Table S1 shows the baseline characteristics of patients starting paliperidone palmitate and oral SGAs. Figure 3 shows the results of the unadjusted and adjusted regression models for: a) no random effects (RE), b) MHSA-level random effects, and c) county-level random effects. After adjustment, the point estimates shifted in the direction of a protective benefit for persons initiating paliperidone palmitate compared to an oral SGA. The adjusted odds ratio (AOR) for hospitalizations was 0.80 (95% CI: 0.59-1.10) and for ED visits was 0.76 (95% CI: 0.56-1.04). Adding MHSA- or county-level random effects moved point estimates slightly toward the null, indicating that individual-level covariates were the most important source of potential bias in this case. The difference between MHSA- and county-level random effects on model results is negligible, suggesting either could be used as a level-2 unit in this analysis. Additional file 1: Tables S2 and S3 present results for these models as well as additional models which included slope-only and intercept and slope random effects.Figure 4 maps the random effects, or deviations from the overall model mean value. P-values indicate if the random effect is significantly different from the mean. Figure 4A and B show county-level and MHSA-level random effects for mental health hospitalizations and indicate one county/MHSA just outside St. Louis where the likelihood of hospitalization is significantly higher than the study area as a whole. Figure 4C and D show random effects for mental health ED visits and indicate a high likelihood of a visit in the same region as well as in St. Louis.

Table 1 shows the variance components estimated from the multi-level models. Overall, about 1% of the variance in hospitalizations and 2% of the variance in ED visits is explained by county or MHSA differences. After adjusting for patient-level factors, this variance is approximately halved, suggesting that underlying population composition factors explain much of the variation across regions. However, since the random effects for adjusted models are statistically significant, there are still area-level contextual factors that are responsible for variation across regions.

Other studies have also suggested that geographic variation in treatments and outcomes can affect CER studies using observational data [7]. Our ESDA results indicated that geographic differences in treatment exposure (drug utilization) and outcomes (mental health hospitalizations and ED visits) are a potential confounder in this comparative effectiveness study of antipsychotics in the Missouri Medicaid system. Therefore, geography was incorporated into the multilevel models. The multilevel models indicated that adjusting for individual-level (compositional) factors explained about half the variation in outcomes evident across counties and MHSAs. This means that unmeasured compositional or additional contextual factors are possibly responsible for variation, for example as shown in the significant random effects in the St. Louis area. If there is sufficient sample, the recommended next step in a CER study would be to include area-level predictors (such as poverty rates or information on provider networks or practice patterns) in the level-2 portion of the multilevel model to better understand (and control for) the area-level contextual effects that drive these differences in outcomes. Due to our small sample size there was insufficient level-2 variance left after adding individual-level variables to continue with this analysis. In this case example, comparing the effectiveness of antipsychotic treatments, patient-level compositional factors were the most important driver of geographic variation. Similar results have been found in a variety of studies across public health disciplines; with many researchers agreeing that area-level characteristics only explain a modest amount of the variation in individual-level outcomes, perhaps in the order of 5-10% [37‐39]. More research is needed to better estimate the range of confounding due to geography in observational comparative effectiveness studies and healthcare utilization outcomes. Exploratory spatial data analysis offers a suite of geographic tools that can be used to examine the potential confounding effect of geographical differences in the treatment and outcome being studied in observational CER studies. The results from ESDA can be used to examine whether geographic clustering conforms to specific geographic boundaries and can inform the choice of the level-2 unit for multilevel regression analyses. In the current study, the SEB rate maps indicated both county-level and MHSA-level patterns of hospitalization.

We suggest the PCORI and AHRQ guidelines could include ESDA and spatial statistical methods as additional techniques useful for statistical analyses of observational CER. In particular, the section describing descriptive statistics and unadjusted analyses could include a discussion of ESDA techniques and the section on adjusted analyses could discuss the use of area-level random effects and area-level predictors within multi-level model frameworks. The PCORI and AHRQ guidelines set high standards for the quality of CER studies and the inclusion of geographic methods will only serve to strengthen these guidelines. Not every study needs to adjust for geospatial variation, but we suggest it should be a CER best practice to test for possible geographic confounding. As most observational datasets commonly used for CER record some type of geographic indicator (e.g., zip code, county, address), conducting geographic analyses should become ubiquitous.

Beyond the utility and importance of the ESDA and spatial statistical analyses presented here, geographically explicit results can assist with the translation of research into practice. State and private health care delivery systems are often geographically defined. Policy makers seek to understand where people who need care are located and whether there are sufficient resources in a given region to support that population. Mapping results, as we do in this study, naturally informs the way in which policy makers intuitively examine the health system. They help to answer questions like: What resources need to be placed where? Which regions of the state have populations that might benefit from introduction of a new drug? Maps and geographic analyses provide valuable tools for hypothesis building and guiding policy decisions.

Study limitations should be acknowledged. First, although many covariates were included in the multivariable modeling, there may be omitted (or unmeasured) factors that are important to the comparisons being made. In addition, because this is an observational comparative effectiveness study in which treatment was not randomized, observed statistical associations do not necessarily mean there is a causal relationship. Therefore, effect estimates should be cautiously interpreted. With regard to this particular comparison, paliperidone palmitate was a newly launched drug at the time of the analysis and the sample size was limited which hindered our ability to find statistical significance. This is particularly problematic for mixed-effect models because small sample sizes within level-2 units lead to non-significant and uninformative random effects estimates. With a larger cohort, we could conduct additional model adjustment, such as propensity score matching or other restricted cohort methods, and then use multi-level models on the matched/restricted cohort. In addition, county- or MHSA-level fixed effects could be used to determine the contextual effects driving significant level-2 random effects. Second, this analysis provides a snapshot in time (one year of new starts) and selection bias and utilization rates may change over time as new drugs are adopted. Future analyses should consider multiple time points in order to examine how results of CER change over time as the new drug is incorporated in to the health system.

In summary, because geographic variation in health care delivery is ubiquitous, unmeasured confounding due to geography may be present in observational CER studies. Because measures of geographic location are commonly available in electronic medical records and administrative claims databases, we argue that CER best practices should include examination of possible geographic confounding in observational data. This paper illustrated methods for conducting such an examination.