Development and validation of a score for evaluating comprehensive stroke care capabilities: J-ASPECT Study

In the first step of the J-ASPECT study, we investigated the current conditions of stroke hospitals in Japan. We created a 49-item questionnaire examining various aspects of stroke care, including medical systems, emergency systems, stroke rehabilitation, education, and medical performance. Some recommended items, such as ventriculostomy availability, were excluded from our questionnaire for the sake of simplicity and to increase the survey response rate since the items seemed identical to the recommendations of board-certified (BC) neurosurgeons in Japan. Other items, such as availability of transesophageal echocardiography, were excluded because of their very low expected usage, which would make an evaluation of their impact on mortality rates difficult. In February 2011, the questionnaire was mailed to 1369 certified training institutions belonging to the Japan Neurosurgical Society, the Japanese Society of Neurology, and the Japan Stroke Society. Based on this questionnaire, the overall organizational and staffing levels of the hospitals, in terms of CSC capacity, were scored following the Brain Attack Coalition recommendations, after reviewing the literature describing CSCs and conducting a thorough discussion with an expert panel [9]. Advanced acute stroke care capabilities were assessed based on 25 items divided into 5 subcategories (listed in Table 1). One point was assigned for each recommended item that the hospital met, resulting in a maximum total score of 25; subcategory scores were also calculated.

Cronbach’s α was calculated to evaluate the consistency between the 5 CSC score subcategories used for assessing CSC capabilities. To determine the influence of each subcategory, α-values were also calculated for all combinations of the four subcategories. Correlations between the 25 CSC score items were determined using tetrachoric correlation coefficients to evaluate individual items measured with different constructs.

Factorial analysis, based on tetrachoric correlation coefficients [10], was performed using principal factor analysis to explore possible potential groupings of the 25 items into a more limited number of components. The selection of the number of components was based on the Eigen values. To understand the meaning of the components, promax rotation was used.

Using the Japanese DPC database for patients hospitalized with strokes during the 2011 fiscal year, we examined the differential effects of the items on mortality and poor outcomes (modified Rankin Scale: 3–6, at discharge) associated with ischemic stroke (IS), intracerebral hemorrhage (ICH), and subarachnoid hemorrhage (SAH). This cross-sectional survey used the DPC discharge database for the institutions participating in the J-ASPECT study. The DPC database is a mixed-case classification system that is linked with the lump-sum payment system, launched in 2002 by the Ministry of Health, Labor and Welfare of Japan [11]. In 2010, approximately 1388 acute care hospitals, representing about 50% of the total hospital beds, had adopted the DPC data system. Data regarding practices were obtained from the DPC database; the attending physician is responsible for each patient’s clinical data entry. The details of this database have been described elsewhere [12].

Of the 749 hospitals that responded to the institutional survey of advanced stroke care capabilities, 256 agreed to participate in the DPC discharge database study. Consecutive patients, hospitalized between April 1, 2010 and March 31, 2011, were identified in the annual discharge database using the International Classification of Diseases (ICD)-10 diagnosis codes related to IS (I63.0-9), nontraumatic ICH (I61.0-9, I62.0-1, I62.9), and SAH (I60.0-9). Patients with scheduled admissions were excluded from analysis. This research was approved by the Institutional Review Board of the National Cerebral and Cardiovascular Center and, if required, by the participating hospitals.

We used hierarchical logistic regression models to determine relationships between hospital CSC scores, reflecting the capacities they were equipped with, and mortality. Each model had two levels of hierarchy (hospital and patient), and considered the random effects of hospital variables as well as the fixed effects of CSC scores, patient age and sex, and Japan Coma Scale (JCS) scores. Interactions such as those between the JCS and CSC scores were not included in the model. The analyses were performed using SAS, version 9.3 (SAS Institute, Cary, NC, USA), and R, version 3.2.0 (R Core Team, R Foundation for Statistical Computing, Vienna, Austria).

First, this study is limited by a possible selection bias because hospitals actively working to improve stroke care were more likely to respond to the questionnaire. However, the coverage of the J-ASPECT Study group, which collaborates with the Japan Neurosurgical Society and the Japanese Congress of Neurological Surgeons, was broad enough to provide a reliable study sample. Second, information bias might have existed (self-reporting, recall, and nonresponse). Third, the CSC score mainly evaluated structural measures and did not consider their utilization, supported with real data. To assess clinical practice quality, the use of process measures is preferred [27], but process measures, such as electrocardiogram monitoring and pulse oximetry, were not considered in this scoring system [4, 28]. However, strong correlations between survey components pertaining to personnel and specific expertise (e.g., availability of neurosurgeons and carotid endarterectomy) were observed in this study, suggesting that these items may not be considered as purely structural, but may have characteristics of both structural and process measures. We are planning to develop a new registry system in the J-ASPECT Study to include key metrics required for certification of CSCs in the US, in addition to the DPC database, to study and monitor the association of such quality metrics on mortality and morbidity of acute stroke patients, in Japan. Fourth, in-hospital mortality was selected as an outcome measure to test the validity of the CSC score. A recent systematic review showed that hospital mortality does not necessarily reflect the quality of clinical practice because mortality is affected to a greater extent by the patients’ condition rather than by the quality of practice [29]. Possible correlations between specific items and mortality in patients with IS may have been missed because of the relatively low in-hospital mortality associated with these patients; a larger study is necessary to resolve this issue. Fifth, the DPC-based payment system contains limited information regarding patient condition severity beyond post-discharge data and the National Institute of Health Stroke (NIHSS) Scale, Glasgow Coma Scale (GCS), ICH-, or Hunt-Hess severity scores, upon admission. Nevertheless, the JCS is a useful tool for evaluating stroke severity. Notably, the importance of the JCS, published in 1974, for predicting stroke outcomes has been recently reconfirmed [9, 30]. Further study is necessary to validate the results of the present study with other patient-level measurements, such as the NIHSS, GCS, etc. Despite the above limitations, clear correlations were revealed between the CSC score and in-hospital mortality in patients with all types of strokes. In future work, the score’s components should be weighted according to stroke type, based on their influence on patient outcomes.