Regional variability in use of a novel assessment of thoracolumbar spine fractures: United States versus international surgeons

Controversy persists with regard to treatment of thoracolumbar injuries. The diagnosis and definition of clinically significant spinal instability remains unclear and poses a source of frequent disagreement in the literature. Some authors note good clinical outcomes with non-operative treatment of these injuries; prospective studies demonstrate that many thoracolumbar fractures may be successfully treated non-operatively, with no benefit gained from adding surgical stabilization [1‐3]. Some patients, however, ultimately fail conservative treatment, developing symptomatic late deformity or instability. Modern operative techniques allow for restoration of normal spinal alignment, correction of instability, and decompression of neural elements. Determining prospectively which patients are prone to developing instability and hence might benefit from surgical treatment remains contentious [4].

Adding to confusion over patient selection, no consensus exists as to choice of treatment in thoracolumbar injuries. The therapeutic approach to these patients is hampered by lack of accepted nomenclature and of a useful and clinically valid classification system for these injuries. While numerous classification systems have been devised, each poses problems in implementation [5]. Many systems are overly complex, limiting their utility. Others omit important portions of standard clinical decision making. Most classification schema fail to suggest treatment options [6].

A recently described treatment algorithm may aid in treatment of these patients. The Thoracolumbar Injury Severity Score (TLISS) assesses injuries based on three criteria: the mechanism of injury based upon radiographic assessment, patient neurological status, and the integrity of the posterior ligamentous complex (PLC) [4]. TLISS was developed by a group of 40 spine experts from 15 trauma centers in the United States, Canada, Australia, Germany, Mexico, France, Sweden, India, and the Netherlands. Relevant literature on thoracolumbar trauma, classification, and treatment was reviewed. A classification scheme and treatment algorithm were described (Figures 1 and 2) [7‐9].

The TLISS algorithm has shown high initial treatment decision validity, with greater than 92% of surveyed surgeons agreeing with the algorithm's treatment recommendation (operative versus nonoperative) [10]. Initial testing showed poor reliability for the injury mechanism sub-score of the measure. Evaluation of early TLISS reliability assessments as well as surveys of the Spine Trauma Study Group led to modification of the protocol, with greater concentration upon injury morphology and de-emphasis of injury mechanism. A separate system, the Thoracolumbar Injury Severity and Classification Score (TLICS), has been recently forwarded [10]. Good validity in different specialties and different training levels have been described, with similarly adequate reliability [2, 11]. Validation across geographic boundaries has not been assessed.

Considerable geographic variation is consistently reported in surgical treatment of spinal disorders. A universal rating scale and treatment algorithm for thoracolumbar injuries must overcome these regional variabilities and demonstrate acceptable reliability and validity regardless of locale. We compare the inter- and intra-rater reliability of the TLISS within and between US and non-US surgeons. We demonstrate moderate to substantial reliability in the use of the scale and high treatment validity, as assessed by surgeon agreement with the algorithm's management recommendation. The TLISS is a promising tool in the evaluation of spine trauma patients.

Forty-eight trauma spine surgeons, including both neurosurgeons and orthopedic surgeons, participated in a comprehensive review and analysis of historical and contemporary thoracolumbar injury classification schemes, eventually leading to development of the TLISS clinical tool [8‐10].

In order to validate the management recommendations of the TLISS algorithm, a booklet of 56 thoracolumbar traumatic injury case studies was prepared. The case vignettes detailed the patient's age, description of the traumatic injury, and neurological exam. Imaging studies, including plain radiographs, CT, and MR (sagittal T₂-weighted images), were included. The cases were distributed to surgeons for classification and grading using the TLISS algorithm. The final severity score was used to determine the recommendation for nonoperative or operative treatment according to the treatment algorithm described in Table 1. Twenty eight surgeons completed the vignettes. The results were analyzed to determine inter-rater reliability, and percent agreement with the final treatment recommendations.

Three months later, the numerical order of the cases was scrambled and pamphlets were redistributed. Twenty-two of the original 48 surgeons who participated in the development of the classification system completed both surveys. Their results were analyzed to determine intra-rater reliability. To assess possible regional differences in the validity and reliability of this system, physicians were grouped into US and non-US cohorts. The US cohort included surgeons from a variety of trauma centers. The international group included surgeons from Canada, Australia, Germany, Mexico, France, Sweden, India, and the Netherlands. The data were then analyzed using SPSS^® and Analyze IT^® software to determine percent agreement, unweighted Cohen's kappa, kappa with linear weighting, and Spearman's rank order correlation. The Cohen's kappa value was defined as the observer agreement (Pa) minus the chance agreement (Pc) divided by the maximum possible agreement that is not related to chance (1-Pc): kappa = (Pa – Pc)/(1-Pc). The kappa values (Table 2) obtained may range from – 1.0 (complete disagreement) through 0 (chance agreement) to 1.0 (perfect agreement) [23]. A guideline for interpreting Cohen's kappa values is summarized in Table 2. For significance tests, all unweighted coefficients were converted into Fisher's z-scores, and the difference in z-scores was divided by standard error. A level was set at 0.05 (ΔZ/SE ≥ 1.96).

Inter- and intra-rater agreement between the cohorts is reviewed in Tables 3 and 4. General results of kappa scoring between the groups has been previously reviewed [10, 12‐14]. Non-US spine surgeons had greater inter-rater reliability on mechanism sub-score (p < 0.05 as assessed by % agreement), while US surgeons had greater inter-rater reliability on neuro status (p < 0.05 as assessed by % agreement and Spearman's r) and PLC integrity (p < 0.05 as assessed by Spearman's r). Intergroup (between USA and international) reliability was similar to within group inter-rater reliability in all parts of the TLISS scoring, indicating that the two groups agreed about as often as individual members within each cohort agreed amongst themselves.

Absolute inter-rater agreement among non-US surgeons on the final TLISS score was greater (p < 0.05), but total TLISS scores better correlated among US surgeons (p < 0.05) (Table 3). With regard to the algorithm's final recommendation for treatment (operative vs. non-operative), inter-rater agreement within the US physician group and non-US group was 75.7% (Cohen's kappa .561) and 72.3% (Cohen's kappa .506), respectively (Table 3). Inter-rater agreement on management between the groups was 74.2% (Cohen's kappa .536). Between the two groups, greatest agreement was found in assessment of neurological status (96.3% agreement, Cohen's kappa .936) (Table 3).

Intra-rater agreement was higher among US surgeons on mechanism, whereas intra-rater agreement was higher among non-US surgeons for neurological status, PLC integrity, and total TLISS score. These differences reached statistical significance as assessed by % agreement for mechanism and neurological status. Differences in intra-rater correlation reached statistical significance on neurological status, PLC, and total TLISS (Table 4). Intra-rater reliability on management in the two cohorts was similar, with 78.8% intra-rater agreement in US surgeons (Cohen's kappa .62) and 77.6% in non-US surgeons (Cohen's kappa .61). US surgeons agreed with the management recommendation of the TLISS in 93.4% of the cases and non-US surgeons agreed with the algorithm in 91.3% of the cases.

Controversy persists in management of thoracolumbar injuries. The TLISS clinical algorithm offers assessment of injury stability and aids in making treatment decisions. The scale has shown adequate reliability between and within different specialties. We show reliability and validity of the TLISS scale across geographic boundaries, comparing US and non-US surgeons. Differences between these broad geographic groups were subtle, with inter-rater reliability between groups similar to reliability within groups. This suggests that the TLISS may help unifying clinical decision making in thoracolumbar trauma.