Next Article in Journal
Safety and Efficacy of Transoral Robotic Thyroidectomy for Thyroid Tumor: A Systematic Review and Meta-Analysis
Next Article in Special Issue
Biological and Clinical Aspects of Metastatic Spinal Tumors
Previous Article in Journal
Safety and Efficacy of Allogeneic Natural Killer Cells in Combination with Pembrolizumab in Patients with Chemotherapy-Refractory Biliary Tract Cancer: A Multicenter Open-Label Phase 1/2a Trial
Previous Article in Special Issue
Prognosis after Palliative Surgery for Patients with Spinal Metastasis: Comparison of Predicted and Actual Survival
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Diagnostic Accuracy of CT for Metastatic Epidural Spinal Cord Compression

by
James Thomas Patrick Decourcy Hallinan
1,2,*,
Shuliang Ge
1,2,
Lei Zhu
3,
Wenqiao Zhang
3,
Yi Ting Lim
1,2,
Yee Liang Thian
1,2,
Pooja Jagmohan
1,2,
Tricia Kuah
1,
Desmond Shi Wei Lim
1,
Xi Zhen Low
1,
Ee Chin Teo
1,
Nesaretnam Barr Kumarakulasinghe
4,
Qai Ven Yap
5,
Yiong Huak Chan
5,
Jiong Hao Tan
6,
Naresh Kumar
6,
Balamurugan A. Vellayappan
7,
Beng Chin Ooi
3,
Swee Tian Quek
1,2 and
Andrew Makmur
1,2
1
Department of Diagnostic Imaging, National University Hospital, 5 Lower Kent Ridge Rd, Singapore 119074, Singapore
2
Department of Diagnostic Radiology, Yong Loo Lin School of Medicine, National University of Singapore, 10 Medical Drive, Singapore 117597, Singapore
3
Department of Computer Science, School of Computing, National University of Singapore, 13 Computing Drive, Singapore 117417, Singapore
4
National University Cancer Institute, NUH Medical Centre (NUHMC), Levels 8–10, 5 Lower Kent Ridge Road, Singapore 119074, Singapore
5
Biostatistics Unit, Yong Loo Lin School of Medicine, 10 Medical Drive, Singapore 117597, Singapore
6
University Spine Centre, Department of Orthopaedic Surgery, National University Health System, 1E, Lower Kent Ridge Road, Singapore 119228, Singapore
7
Department of Radiation Oncology, National University Cancer Institute Singapore, National University Hospital, Singapore 119074, Singapore
*
Author to whom correspondence should be addressed.
Cancers 2022, 14(17), 4231; https://doi.org/10.3390/cancers14174231
Submission received: 1 August 2022 / Revised: 23 August 2022 / Accepted: 25 August 2022 / Published: 31 August 2022

Abstract

:

Simple Summary

Early diagnosis of metastatic epidural spinal cord compression (MESCC) is vital to prevent paralysis. Staging CT scans are performed routinely in cancer patients and could detect MESCC earlier. In this study, we assessed the performance of the original radiologist report for MESCC grading compared to three radiologists performing detailed MESCC evaluation using dedicated CT windows. Two expert radiologists provided the reference standard using MRI scans performed within 30 days. For normal/none versus low/high-grade MESCC per CT scan, all radiologists demonstrated almost perfect agreement with kappa values ranging from 0.866 (95% CI 0.787–0.945) to 0.947 (95% CI 0.899–0.995), compared to only slight agreement for the reports (kappa = 0.095, 95% CI−0.098–0.287). Radiologists also showed high sensitivities ranging from 91.51 (95% CI 84.49–96.04) to 98.11 (95% CI 93.35–99.77), compared to 44.34 (95% CI 34.69–54.31) for the reports. In conclusion, a dedicated radiologist review for MESCC on CT showed improved performance compared to the original report (current standard of care).

Abstract

Background: Early diagnosis of metastatic epidural spinal cord compression (MESCC) is vital to expedite therapy and prevent paralysis. Staging CT is performed routinely in cancer patients and presents an opportunity for earlier diagnosis. Methods: This retrospective study included 123 CT scans from 101 patients who underwent spine MRI within 30 days, excluding 549 CT scans from 216 patients due to CT performed post-MRI, non-contrast CT, or a gap greater than 30 days between modalities. Reference standard MESCC gradings on CT were provided in consensus via two spine radiologists (11 and 7 years of experience) analyzing the MRI scans. CT scans were labeled using the original reports and by three radiologists (3, 13, and 14 years of experience) using dedicated CT windowing. Results: For normal/none versus low/high-grade MESCC per CT scan, all radiologists demonstrated almost perfect agreement with kappa values ranging from 0.866 (95% CI 0.787–0.945) to 0.947 (95% CI 0.899–0.995), compared to slight agreement for the reports (kappa = 0.095, 95%CI −0.098–0.287). Radiologists also showed high sensitivities ranging from 91.51 (95% CI 84.49–96.04) to 98.11 (95% CI 93.35–99.77), compared to 44.34 (95% CI 34.69–54.31) for the reports. Conclusion: Dedicated radiologist review for MESCC on CT showed high interobserver agreement and sensitivity compared to the current standard of care.

1. Introduction

Vertebral metastases are seen in up to 40% of patients with cancer, and approximately 20% of such patients progress to metastatic epidural spinal cord compression (MESCC) [1]. Early diagnosis of MESCC is vital to initiate appropriate therapy and prevent permanent neurological dysfunction [2,3]. In addition, with the continuing development of more effective and targeted systemic therapies, the incidence of vertebral metastases and their complications are expected to increase [4,5,6,7].
The spinal column is the most common site for bone metastases, and the most frequent sites of MESCC are along the thoracic vertebrae (70%), lumbar vertebrae (20%), and cervical vertebrae (10%) [8]. Clinical diagnosis of MESCC can be challenging as patients may be asymptomatic or present with non-specific symptoms, which can overlap with those already present from degenerative spine disease. Symptoms preceding paresis or paralysis in MESCC include pain (change in frequency or characteristics), reduced mobility, and altered sensations (e.g., temperature sensations or tingling). Given these clinical factors, confirmatory imaging diagnosis of MESCC can be hampered [9,10].
MRI is considered the most accurate imaging modality for the classification of MESCC [11,12]. MRI can provide excellent visualization of vertebral metastatic involvement and any associated malignant epidural disease and its impact on the cerebrospinal fluid space and spinal cord [13]. In cases where MRI is contraindicated, a CT myelogram is another accurate modality for assessing the degree of MESCC. Axial T2-weighted MRI images most accurately characterize the extent of MESCC via the Bilsky grading scale developed by the Spine Oncology Study Group (SOSG) [14,15]. The grading scale uses six groupings, which can be further divided into two key clinical categories; low-grade MESCC (grades 0, 1a, or 1b), which can be treated with radiotherapy (including stereotactic body radiotherapy, SBRT), and high-grade MESCC (grades 1c, or 2, or 3) which can be considered for initial surgery (e.g., separation surgery) and subsequent radiotherapy [16,17,18].
Other potential imaging modalities for MESCC include using contrast-enhanced CT without dedicated myelography or spinal protocols. Cancer patients frequently undergo CT studies of the chest and abdomen to assess the overall metastatic disease burden and response to treatment. This presents an opportunity for expedited diagnosis of MESCC and impending cord compression in asymptomatic patients or those with unclear symptoms on a background of pre-existing pain and high-dose analgesics [19]. Prior studies have assessed the utility of CT for assessing MESCC in symptomatic patients. Crocker et al. (2011) evaluated the use of staging CT in patients with suspected MESCC to identify those requiring transfer to a specialist neuroscience center for MRI [20]. In their small study of forty-four patients, CT had a sensitivity/specificity of 89%/92% for MESCC compared to the eventual MRI. Currently, the NICE guidelines on MESCC in adult patients (2008) do not recommend staging CT as a screening tool for MESCC. However, the studies analyzed did not use the latest CT scanners, and the guidelines are due to be updated soon [12,21].
This study aims to retrospectively investigate the performance of routine staging CT without myelography to detect MESCC in a group of patients who underwent MRI for suspected MESCC. CT scans performed within 30 days of the MRI will be assessed to reduce the potential for disease progression. The reference standard on CT will be provided by experienced spine radiologists using the corresponding MRI. The accuracy of CT will be determined using the existing radiology reports for the CT scans, along with a dedicated independent assessment of the scans by board-certified radiologists specializing in oncological imaging.

2. Materials and Methods

This study was approved by the National University Hospital, Singapore ethics review board and complied with the Health Insurance Portability and Accountability Act (HIPAA). Waiver of patient informed consent was allowed as there was minimal patient risk involved.

2.1. CT Scan Dataset Preparation

Retrospective retrieval and anonymization of CT studies of the thorax and upper abdomen and corresponding MRI spines from patients with known spinal column metastatic disease and suspicion of MESCC was undertaken over fourteen years from September 2007 through September 2021 at the National University Hospital, Singapore. Adults (≥18 years) were included with imaging examinations obtained across different MRI scanners (General Electric and Siemens) and CT platforms (Philips, General Electric, and Siemens platforms). Staging CT scans and corresponding MRI spine studies with a time gap of up to 1 month (30 days) were included. CT scans with intravenous contrast were used, excluding CT scans performed without contrast. In addition, staging CT scans with no corresponding MRI study (no reference standard) or CT scans performed post-MRI were excluded. MESCC on MRI was assessed using axial T2-weighted images for comparison with the staging CT. Table 1 and Table 2 provide details for the CT and MRI scanner subtypes and imaging parameters.

2.2. CT Scan Labeling

The staging CT dataset was manually labeled for MESCC in consensus by board-certified spine imaging specialists (AM with 7 years of experience and JTPDH with 11 years of experience) and served as the reference standard. The two radiologists labeled each axial CT image in consensus with the corresponding MRI (performed within one month) using an established visual Bilsky SOSG scale (The visual scale is provided in Figure 1).
To compare with the imaging reference standard and assess diagnostic accuracy and interobserver variability, the CT dataset was labeled independently using standardized CT windowing by three body radiologists interested in oncology imaging (YT; 3 years, PJ; 14 years, and YLT with 13 years of experience). Prior to labeling, all body radiologists were provided with a visual MESCC grading scale and reviewed practice studies providing examples of low and high-grade MESCC on MRI, and CT scans side by side. For the official focused imaging assessment of MESCC, all body radiologists were blinded to the reference standard MESCC gradings, original reports, and corresponding MRI scans.
Labeling was performed using an open-source image labeling software (LabelImg—https://github.com/tzutalin/labelImg) (accessed on 30 May 2022). This program allowed bounding boxes to be placed on each axial contrast-enhanced CT image to segment the region of evaluation around the spinal canal (C7-T1 through to the conus at T12-L3). The radiologist could click on the bounding box and change the MESCC grade. The axial CT images were evaluated using three common Hounsfield unit (HU) windows widths (W) and levels (L) for optimal radiologist assessment of MESCC; Abdomen/pelvis soft tissue windowing (W:400, L:50), bone windowing (W:1500, L:300), and additional spine soft tissue windowing (W:200, L:100). These are provided as standard settings on the GE centricity PACS (Picture archiving and communication system).
When inspecting each axial CT image, the radiologists classified MESCC using a three-grade scheme; Normal (no discernible epidural disease or Bilsky grade 0), Bilsky grades 1a or 1b amenable to first-line radiotherapy (low-grade MESCC), and grades 1c or 2 or 3, which more frequently require primary surgical decompression (high-grade MESCC) [14,16]. The visual MESCC grading scale provided to all the radiologists is shown in Figure 1. Degenerative changes (disk bulges, osteophytic change, and ligamentum flavum redundancy/ossification) producing spinal canal narrowing/stenosis were labeled and excluded from further analysis [22,23].

2.3. Staging CT Radiology Report Review

Along with the focused assessment for MESCC by three radiologists, we compared the accuracy of the original radiology report to the reference standard. A review of the radiology reports for each staging CT study was undertaken to assess whether there was documentation of any grade of MESCC per CT scan and at which site along the thoracic spine. Along with documentation of the MESCC grade, other terms used in the report to document MESCC included “encroachment of the spinal canal”, “Epidural disease is noted at.…”, “extension of disease into the spinal canal”, and “enhancing soft tissue noted in the spinal canal at.…”.

2.4. Statistical Analysis

All analyses were performed using Stata version 16 (StataCorp) with statistical significance set at 2-sided p < 0.05. Descriptive statistics for continuous variables were presented as mean ± SD (range) and n (%) for categorical variables. Inter-observer agreement was also calculated per axial CT image and per CT scan for both three grade (normal/no disease, low-grade, and high-grade MESCC) and two grade (normal/low-grade versus high-grade MESCC, and normal versus low/high-grade MESCC) classification using Gwet’s kappa to take into consideration a high percentage of normal/grade 0 classification [24,25]. A comparison of the radiology reports with the reference standard and radiologists using two-grade MESCC classification per CT scan (normal/low versus high-grade MESCC and normal versus low/high-grade MESCC classification) was performed. Sensitivity, specificity along with AUCs were calculated for two-grade MESCC classification.
Levels of interobserver agreement for each Gwet’s kappa value are classified as <0 = poor, 0–0.2 = slight, 0.21–0.4 = fair, 0.41–0.6 = moderate, 0.61–0.8 = substantial, and 0.81–1 = almost-perfect agreement. Furthermore, 95% confidence intervals (CIs) were presented.

3. Results

3.1. Patient Demographics, Cancer Subtypes, and MESCC Sites

An imaging data review over the fourteen-year study identified 369 patients who underwent a spine MRI for suspected thoracic MESCC. Of these, 52/369 patients (14.1%) did not have any CT study performed and were excluded from further analysis. For the remaining 317/369 patients (85.9%), there were 672 CT studies identified. Of these, 549/672 (81.7%) CT studies were excluded due to the following reasons: CT performed post-MRI (313/549, 57.0%), greater than a 30-day gap between the CT and MRI (131/549, 23.9%), CT without intravenous contrast (24/549, 4.4%), and CT images unavailable for extraction (81/549, 14.8%). Overall, there were 123 CT studies available from 101 patients for detailed analysis.
Overall, the mean age, in years, of the 101 patients was 60 ± 11.6 (SD) (range: 26–93 years). Males (54/101 patients, 53.5%) made up a greater proportion of the group than females (47/101 patients, 46.5%). Lung carcinoma (29/101 patients, 28.7%) was the most frequent malignancy, followed by breast (21/101 patients, 20.8%) and colorectal (12/101 patients, 11.9%) carcinomas. The most frequent site of MESCC along the thoracic spine was at the junction with the lumbar spine between T11-L3 (39/123 CT scans, 31.7%). Full details on the patient cancer subtypes and the MESCC locations along the spine are documented in Table 3. Figure 2 provides a study flow chart for the selection of cases.

3.2. Reference Standard

The number of axial CT images assessed and the corresponding MESCC grade is highlighted in Table 4. There was a high proportion of normal or grade 0 MESCC gradings for 5642/6545 images (86.2%). Low or high-grade MESCC gradings accounted for 903/6545 images (13.8%), with high-grade disease accounting for 432/903 (47.8%) images with MESCC.
On a per CT scan basis, 90/123 (73.1%) studies had at least one site of high-grade MESCC, 16/123 scans (13.0%) had at least one site of low-grade MESCC with no high-grade lesion, and 17/123 scans (13.8%) were classified as normal/grade 0 with no site of low or high-grade MESCC.

3.3. Three-Grade MESCC Classification

The interobserver agreements for all radiologists against the reference standard are shown in Table 5. For three-grade MESCC classification per CT image, all radiologists showed almost perfect agreement with the reference standard, with kappa values ranging from 0.927 (PJ, 95% CI 0.920–0.934) to 0.931 (YT, 95% CI 0.924–0.938). For three-grade classification per CT scan, two of the radiologists showed substantial agreement with kappa values of 0.692 (PJ, 95% CI 0.585–0.798) and 0.694 (YT, 95% CI 0.587–0.801), and TYL showed almost perfect agreement with a kappa of 0.854 (95% CI 0.777–0.931). MESCC examples on CT and MRI are shown in Figure 3 and Figure 4.

3.4. Two-Grade (Normal/Low Versus High-Grade) MESCC Classification

All radiologists showed almost perfect agreement for two-grade normal/low versus high-grade MESCC per CT image with kappa values ranging from 0.967 (PJ, 95% CI 0.962–0.972) to 0.978 (TYL, 95% CI 0.974–0.982) (Table 6).
Per CT image, the radiologists (PJ, TYL, and YT) showed high specificities (all >99%), whereas the sensitivities ranged from 56.48 (PJ, 95% CI 51.66–61.21) to 77.78 (TYL, 95% CI 73.56–81.61).
Per CT scan, TYL showed almost perfect agreement with a kappa of 0.905 (95% CI 0.833–0.976), while PJ and YT showed substantial agreement with kappa values of 0.653 (95% CI 0.516–0.789) and 0.688 (95% CI 0.558–0.819), respectively. The original radiology reports only had a fair interobserver agreement with a kappa of 0.213 (95% CI 0.036–0.391), which was significantly reduced compared to all radiologists (all p < 0.001) (Table 7).
Per the CT scan, the radiologists showed high specificities ranging from 93.94 (TYL, 95% CI 79.77–99.26) to 100 (PJ, 95% CI 89.42–100.00). Per CT scan sensitivity ranged from 74.44 (PJ, 95% CI 64.16–83.06) to 94.44 (TYL, 95% CI 87.51–98.17). Compared to the original reports, the sensitivities for all radiologists were superior, e.g., 74.44 (PJ, 95% CI 64.16–83.06) compared to only 48.89 (95% CI 38.20–59.65) for the reports (all p < 0.001). The AUCs for all the radiologists were also superior to the original reports, e.g., the lowest radiologist AUC of 0.872 (PJ, 95% CI 0.827–0.918), compared to 0.699 (95% CI 0.627–0.771) for the reports (all p < 0.001).

3.5. Two-Grade (Normal versus Low/High-Grade) MESCC Classification

All radiologists showed high (almost-perfect) interobserver agreement for normal/none versus low/high-grade MESCC per CT image with kappa values ranging from 0.936 (TYL, 95% CI 0.929–0.943) to 0.949 (PJ, 95% CI 0.943–0.955) (Table 8). Similar results (almost perfect agreement) were seen per CT scan with kappa values ranging from 0.866 (YT, 95% CI 0.787–0.945) to 0.947 (PJ, 95% CI 0.899–0.995) (Table 9). Per CT scan, the original radiology reports had only slight interobserver agreement with a kappa of 0.095 (95% CI −0.098–0.287, p = 0.333), which was significantly less compared to all radiologists (all p < 0.001).
Per CT image, the radiologists (PJ, TYL, and YT) showed high specificities (range 97.5–99.03), and the sensitivities ranged from 73.31 (TYL, 95% CI 70.30–76.17) to 86.82 (YT, 95% CI 84.44–88.96) (Table 8).
Per CT scan, the radiologists showed high sensitivities ranging from 91.51 (YT, 95% CI 84.49–96.04) to 98.11 (PJ and TYL, both 95% CIs 93.35–99.77). Specificities for detecting low/high-grade MESCC ranged from 64.71 (TYL, 95% CI 38.33–85.79) to 82.35 (PJ and YT, both 95% CIs 56.57–96.20). Compared to the original reports, the sensitivities for all radiologists were superior, e.g., the lowest sensitivity of 91.51 (YT, 95% CI 84.49–96.04) compared to only 44.34 (95% CI 34.69–54.31) for the reports (all p < 0.001). The AUCs for PJ and YT were superior to the original reports, with AUC of 0.902 (95% CI 0.808–0.997) for PJ and AUC of 0.869 (95% CI 0.772–0.966) for YT compared to 0.722 (95% CI 0.674–0.769) for the reports (p < 0.001 and p = 0.008, respectively) (Table 9).

4. Discussion

Early diagnosis of metastatic epidural spinal cord compression (MESCC) is important to expedite therapy and prevent paralysis. Staging CT scans are performed routinely in cancer patients and present a window of opportunity for earlier MESCC diagnosis, allowing for detailed MRI evaluation in selected patients. In this study, we assessed the performance of the original radiologist CT report for MESCC grading compared to three radiologists who performed detailed MESCC evaluations using dedicated CT windows. Two expert radiologists provided the reference standard using the corresponding MRI study performed within 30 days of the CT. For normal/none versus low/high-grade MESCC per CT scan, all radiologists demonstrated almost perfect interobserver agreement with kappa values ranging from 0.866 (95% CI 0.787–0.945) to 0.947 (95% CI 0.899–0.995), compared to only slight interobserver agreement for the original reports (kappa = 0.095, 95% CI−0.098–0.287, p = 0.333). Radiologists also showed high sensitivities ranging from 91.51 (95% CI 84.49–96.04) to 98.11 (95% CI 93.35–99.77), compared to only 44.34 (95% CI 34.69–54.31) for the reports.
Prior studies have shown the potential of CT to detect MESCC [20,26,27]. Pezaro et al. (2015) assessed 34 patients with known metastatic prostatic carcinoma for evidence of MESCC on staging CT [27]. They noted that MESCC was already detectable in 28/34 (80%) of the patients a median of 28 days prior to definitive spine MRI. Another study by Crocker et al. (2011) assessed the use of CT to screen for suspected MESCC to determine whether the patient required transfer to a specialist center for definitive MRI [20]. In their study, contrast-enhanced CT had a sensitivity of 89% with a high specificity of 92% for MESCC in 44 patients compared to the eventual MRI. This compares favorably to our more extensive study of 101 patients, with dedicated radiologist review showing high sensitivity of up to 98.11 and specificity of up to 82.35 for two-grade normal/none versus low/high-grade MESCC.
Accurate detection of MESCC on CT has several advantages. First, earlier detection of MESCC on routine staging CT could reduce the time to definitive MRI and treatment with improved patient outcomes, including a reduced risk of irreversible neurological injury. Suppose MESCC is detected at an earlier Bilsky grade (e.g., 1a or 1b). In that case, there is also an option for initial less invasive systemic treatment or radiotherapy, which could reduce the need for surgical intervention. Second, patients with suspected MESCC but limited access to MRI in underserved regions could undergo initial screening CT to determine the need for urgent transfer to a specialist center.
In our study, the original radiology CT reports had only slight interobserver agreement (kappa = 0.095) and reduced sensitivity (44.34) for detecting any grade of MESCC compared to a dedicated review. This poor performance is likely multifactorial. Firstly, on staging CT studies (without clinical suspicion of MESCC), radiologists will have to interrogate multiple regions, including the lungs, mediastinum, solid viscera (e.g., liver), bowel, and bony structures, including the spine. Secondly, dedicated bone windowing is typically used to assess for fractures and destructive bone metastases but does not provide adequate evaluation of the epidural space or paravertebral regions. This study’s combination of dedicated bone and soft tissue windows provided optimal assessment for any destructive bony lesion and associated epidural mass [28].
Our study has several limitations. First, we did not have multiplanar coronal and sagittal reconstructions for the CT studies and assessed the accuracy of MESCC classification using axial images alone. The use of multiplanar reconstructions, especially sagittal images similar to MRI, could improve the accuracy of MESCC assessment. Second, only CT scans with a spine MRI conducted within 1 month were analyzed to limit the potential disease progression between the studies. This limited the assessment of any time delays between the CT and MRI studies, which may have resulted in delayed MESCC diagnosis. Future studies could assess the utility of CT scans to detect epidural disease prior to the MRI using a more extended time period (e.g., up to 1 year). Expedited diagnosis of MESCC on CT could allow for earlier administration of less invasive systemic treatment or radiotherapy, reducing the need for surgical intervention [29,30]. Third, the CT scans used for analysis were identified using a surgical MESCC database. This resulted in an artificially higher proportion of positive cases with MESCC and may not represent actual day-to-day clinical practice. A potential concern is MESCC being overcalled on staging CT leading to an increase in unnecessary MRI studies and increased healthcare costs.

5. Conclusions

In conclusion, we assessed the utility of staging CT for the classification of MESCC. Staging CT represents a window of opportunity to triage patients for further definitive MRI. An earlier diagnosis of MESCC could allow the initiation of less invasive therapy to prevent costly surgical intervention. Radiologists using dedicated CT windows showed high interobserver agreement (kappa values ranging from 0.866 to 0.947) and sensitivities (ranging from 91.51 to 98.11) for recognition of any grade of MESCC. The original radiology reports showed relatively reduced performance (kappa = 0.095, sensitivity = 44.34), suggesting that dedicated training sessions for assessment of MESCC on CT or even the use of a deep learning algorithm for automatic classification of MESCC could improve patient care [31,32]. Future prospective studies are planned to assess the accuracy of MESCC detection on staging CT across a more extensive range of oncological patients.

Author Contributions

Conceptualization, J.T.P.D.H., N.B.K., J.H.T., N.K., B.A.V., S.T.Q. and A.M.; Data curation, J.T.P.D.H., S.G., L.Z., W.Z., T.K., D.S.W.L., X.Z.L., E.C.T. and A.M.; Formal analysis, J.T.P.D.H., S.G., L.Z., W.Z., Y.T.L., Y.L.T., P.J., Q.V.Y., Y.H.C. and B.A.V.; Funding acquisition, J.T.P.D.H., E.C.T., Y.H.C., J.H.T., N.K., S.T.Q. and A.M.; Investigation, J.T.P.D.H., S.G., Y.T.L., Y.L.T., P.J., T.K., D.S.W.L., X.Z.L., Q.V.Y. and A.M.; Methodology, J.T.P.D.H., S.G., L.Z., N.B.K., Q.V.Y., Y.H.C., J.H.T., B.A.V., B.C.O., S.T.Q. and A.M.; Project administration, J.T.P.D.H., T.K., D.S.W.L., E.C.T., N.K. and A.M.; Resources, J.T.P.D.H., S.G., L.Z., W.Z., Y.L.T., T.K., D.S.W.L., X.Z.L., E.C.T., N.B.K., Q.V.Y., Y.H.C., J.H.T., N.K., B.C.O., S.T.Q. and A.M.; Software, J.T.P.D.H., S.G., L.Z., W.Z., Y.T.L. and B.C.O.; Supervision, J.T.P.D.H., E.C.T., Y.H.C., J.H.T., N.K., B.A.V. and S.T.Q.; Validation, L.Z., W.Z., Y.T.L., Y.L.T., P.J. and B.C.O.; Visualization, J.T.P.D.H., S.G., Y.T.L., Y.L.T., P.J., T.K., D.S.W.L., X.Z.L. and A.M.; Writing—original draft, J.T.P.D.H., S.G., Q.V.Y., J.H.T. and B.A.V.; Writing—review and editing, J.T.P.D.H., N.B.K., N.K., B.A.V. and A.M. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by: (1) Direct Funding from MOH/NMRC: This research is supported by the Singapore Ministry of Health National Medical Research Council under the NMRC Clinician-scientist individual research grant, new investigator grant (CS-IRG NIG); Grant Title: Deep learning pathway for the management of spine metastases (CNIG20nov-0011, MOH-000725). (2) NCIS Centre Grant Seed Funding Program (December 2020 Grant Call); Grant Title: Artificial Intelligence for the management of vertebral metastases.

Institutional Review Board Statement

The study was conducted in accordance with the Declaration of Helsinki and approved by the Institutional Review Board of the National Healthcare Group (NHG), Singapore (protocol code NHG DSRB Ref: 2020/00835 on 17 September 2021).

Informed Consent Statement

This study was approved by our institutional review board and compliant with the Health Insurance Portability and Accountability Act (HIPAA). Waiver of consent was granted due to the retrospective nature of the study and minimal risk involved.

Data Availability Statement

The data presented in this study are available on request from the corresponding author.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

References

  1. Chiu, R.G.; Mehta, A.I. Spinal Metastases. JAMA 2020, 323, 2438. [Google Scholar] [CrossRef] [PubMed]
  2. Cole, J.S.; Patchell, R.A. Metastatic epidural spinal cord compression. Lancet Neurol. 2008, 7, 459–466. [Google Scholar] [CrossRef]
  3. Nowak, H.; Szwacka, D.M.; Pater, M.; Mrugalski, W.K.; Milczarek, M.G.; Staniszewska, M.; Jankowski, R.; Barciszewska, A.M. Holistic Approach to the Diagnosis and Treatment of Patients with Tumor Metastases to the Spine. Cancers 2022, 14, 3480. [Google Scholar] [CrossRef] [PubMed]
  4. Barzilai, O.; Fisher, C.G.; Bilsky, M.H. State of the Art Treatment of Spinal Metastatic Disease. Neurosurgery 2018, 82, 757–769. [Google Scholar] [CrossRef] [PubMed]
  5. Joaquim, A.F.; Powers, A.; Laufer, I.; Bilsky, M.H. An update in the management of spinal metastases. Arq. Neuropsiquiatr. 2015, 73, 795–802. [Google Scholar] [CrossRef]
  6. Laur, O.; Nandu, H.; Titelbaum, D.S.; Nunez, D.B.; Khurana, B. Nontraumatic Spinal Cord Compression: MRI Primer for Emergency Department Radiologists. Radiographics 2019, 39, 1862–1880. [Google Scholar] [CrossRef]
  7. Spratt, D.E.; Beeler, W.H.; de Moraes, F.Y.; Rhines, L.D.; Gemmete, J.J.; Chaudhary, N.; Shultz, D.B.; Smith, S.R.; Berlin, A.; Dahele, M.; et al. An integrated multidisciplinary algorithm for the management of spinal metastases: An International Spine Oncology Consortium report. Lancet Oncol. 2017, 18, e720–e730. [Google Scholar] [CrossRef]
  8. Mundy, G.R. Metastasis to bone: Causes, consequences and therapeutic opportunities. Nat. Rev. Cancer 2002, 2, 584–593. [Google Scholar] [CrossRef]
  9. Guzik, G. Analysis of factors delaying the surgical treatment of patients with neurological deficits in the course of spinal metastatic disease. BMC Palliat. Care 2018, 17, 44. [Google Scholar] [CrossRef]
  10. Patel, D.A.; Campian, J.L. Diagnostic and Therapeutic Strategies for Patients with Malignant Epidural Spinal Cord Compression. Curr. Treat. Options Oncol. 2017, 18, 53. [Google Scholar] [CrossRef]
  11. Shah, L.M.; Salzman, K.L. Imaging of spinal metastatic disease. Int. J. Surg. Oncol. 2011, 2011, 769753. [Google Scholar] [CrossRef] [PubMed]
  12. National Institute for Health and Care Excellence. Metastatic Spinal Cord Compression: Diagnosis and Management of Adults at Risk of and with Metastatic Spinal Cord Compression NICE Guidelines (CG75); NICE: London, UK, 2008. [Google Scholar]
  13. Vellayappan, B.A.; Kumar, N.; Chang, E.L.; Sahgal, A.; Sloan, A.E.; Lo, S.S. Novel multidisciplinary approaches in the management of metastatic epidural spinal cord compression. Future Oncol. 2018, 14, 1665–1668. [Google Scholar] [CrossRef]
  14. Bilsky, M.H.; Laufer, I.; Fourney, D.R.; Groff, M.; Schmidt, M.H.; Varga, P.P.; Vrionis, F.D.; Yamada, Y.; Gerszten, P.C.; Kuklo, T.R. Reliability analysis of the epidural spinal cord compression scale. J. Neurosurg. Spine 2010, 13, 324–328. [Google Scholar] [CrossRef] [PubMed]
  15. Liu, J.K.; Laufer, I.; Bilsky, M.H. Update on management of vertebral column tumors. CNS Oncol. 2014, 3, 137–147. [Google Scholar] [CrossRef] [PubMed]
  16. Laufer, I.; Rubin, D.G.; Lis, E.; Cox, B.W.; Stubblefield, M.D.; Yamada, Y.; Bilsky, M.H. The NOMS framework: Approach to the treatment of spinal metastatic tumors. Oncologist 2013, 18, 744–751. [Google Scholar] [CrossRef] [PubMed]
  17. Quraishi, N.A.; Arealis, G.; Salem, K.M.; Purushothamdas, S.; Edwards, K.L.; Boszczyk, B.M. The surgical management of metastatic spinal tumors based on an Epidural Spinal Cord Compression (ESCC) scale. Spine J. 2015, 15, 1738–1743. [Google Scholar] [CrossRef] [PubMed]
  18. Barzilai, O.; Laufer, I.; Yamada, Y.; Higginson, D.S.; Schmitt, A.M.; Lis, E.; Bilsky, M.H. Integrating Evidence-Based Medicine for Treatment of Spinal Metastases into a Decision Framework: Neurologic, Oncologic, Mechanicals Stability, and Systemic Disease. J. Clin. Oncol. 2017, 35, 2419–2427. [Google Scholar] [CrossRef]
  19. Switlyk, M.D.; Hole, K.H.; Skjeldal, S.; Hald, J.K.; Knutstad, K.; Seierstad, T.; Zaikova, O. MRI and neurological findings in patients with spinal metastases. Acta Radiol. 2012, 53, 1164–1172. [Google Scholar] [CrossRef]
  20. Crocker, M.; Anthantharanjit, R.; Jones, T.L.; Shoeb, M.; Joshi, Y.; Papadopoulos, M.C.; Bell, B.A.; Rich, P. An extended role for CT in the emergency diagnosis of malignant spinal cord compression. Clin. Radiol. 2011, 66, 922–927. [Google Scholar] [CrossRef]
  21. National Institute for Health and Care Excellence. 2019 Surveillance of Metastatic Spinal Cord Compression in Adults: Risk Assessment, Diagnosis and Management (NICE Guideline CG75) [Internet]; National Institute for Health and Care Excellence (NICE): London, UK, 2019. Available online: https://www.ncbi.nlm.nih.gov/books/NBK550623/ (accessed on 30 June 2022).
  22. Lurie, J.D.; Tosteson, A.N.; Tosteson, T.D.; Carragee, E.; Carrino, J.A.; Kaiser, J.; Sequeiros, R.T.; Lecomte, A.R.; Grove, M.R.; Blood, E.A.; et al. Reliability of readings of magnetic resonance imaging features of lumbar spinal stenosis. Spine 2008, 33, 1605–1610, Erratum in Spine 2008, 33, 2482. [Google Scholar] [CrossRef]
  23. Fardon, D.F.; Williams, A.L.; Dohring, E.J.; Murtagh, F.R.; Gabriel Rothman, S.L.; Sze, G.K. Lumbar disc nomenclature: Version 2.0: Recommendations of the combined task forces of the North American Spine Society, the American Society of Spine Radiology and the American Society of Neuroradiology. Spine J. 2014, 14, 2525–2545. [Google Scholar] [CrossRef]
  24. Gwet, K.L. Computing inter-rater reliability and its variance in the presence of high agreement. Br. J. Math. Stat. Psychol. 2008, 61 Pt 1, 29–48. [Google Scholar] [CrossRef] [PubMed]
  25. Honda, C.; Ohyama, T. Homogeneity score test of AC1 statistics and estimation of common AC1 in multiple or stratified inter-rater agreement studies. BMC Med. Res. Methodol. 2020, 20, 20. [Google Scholar] [CrossRef] [PubMed]
  26. Catherine, G.; MacLeod, N.; Sheridan, S. Is CT adequate to assess for malignant cord compression? Abstract Only. Clin. Radiol. 2015, 70 (Suppl. S1), s7. [Google Scholar] [CrossRef]
  27. Pezaro, C.; Omlin, A.; Perez-Lopez, R.; Mukherji, D.; Attard, G.; Bianchini, D.; Lorente, D.; Parker, C.; Dearnaley, D.; de Bono, J.S.; et al. Progressive computed tomography (CT) appearances preceding malignant spinal cord compression (MESCC) in men with castration-resistant prostate cancer. Clin. Radiol. 2015, 70, 359–365. [Google Scholar] [CrossRef] [PubMed]
  28. Kuah, T.; Vellayappan, B.A.; Makmur, A.; Nair, S.; Song, J.; Tan, J.H.; Kumar, N.; Quek, S.T.; Hallinan, J.T.P.D. State-of-the-Art Imaging Techniques in Metastatic Spinal Cord Compression. Cancers 2022, 14, 3289. [Google Scholar] [CrossRef] [PubMed]
  29. Robson, P. Metastatic spinal cord compression: A rare but important complication of cancer. Clin. Med. 2014, 14, 542–545. [Google Scholar] [CrossRef]
  30. Boussios, S.; Cooke, D.; Hayward, C.; Kanellos, F.S.; Tsiouris, A.K.; Chatziantoniou, A.A.; Zakynthinakis-Kyriakou, N.; Karathanasi, A. Metastatic Spinal Cord Compression: Unraveling the Diagnostic and Therapeutic Challenges. Anticancer Res. 2018, 38, 4987–4997. [Google Scholar] [CrossRef]
  31. Hallinan, J.T.P.D.; Zhu, L.; Zhang, W.; Lim, D.S.W.; Baskar, S.; Low, X.Z.; Yeong, K.Y.; Teo, E.C.; Kumarakulasinghe, N.B.; Yap, Q.V.; et al. Deep Learning Model for Classifying Metastatic Epidural Spinal Cord Compression on MRI. Front. Oncol. 2022, 12, 849447. [Google Scholar] [CrossRef]
  32. Hallinan, J.T.P.D.; Zhu, L.; Zhang, W.; Kuah, T.; Lim, D.S.W.; Low, X.Z.; Cheng, A.J.L.; Eide, S.E.; Ong, H.Y.; Muhamat Nor, F.E.; et al. Deep Learning Model for Grading Metastatic Epidural Spinal Cord Compression on Staging CT. Cancers 2022, 14, 3219. [Google Scholar] [CrossRef]
Figure 1. Bilsky grading for metastatic epidural spinal cord compression (MESCC) was demonstrated using axial CT, MRI (T2-weighted), and a picture for each grade (right to left). The red shaded region shows MESCC in each image. Grade 0/normal (Black outline): No metastatic epidural disease present. Low-grade MESCC (Blue outline); Grade 1a: Metastatic epidural soft tissue with no thecal sac deformity, or Grade 1b: Metastatic epidural soft tissue with thecal sac deformity but no spinal cord contact. High-grade MESCC (Red outline); Grade 1c: Metastatic epidural soft tissue touching the spinal cord with no discernible compression or displacement, or Grade 2: Metastatic epidural soft tissue cord compression with some surrounding cerebrospinal fluid still visible, or Grade 3: Metastatic epidural spinal cord compression without discernible surrounding cerebrospinal fluid. Thecal sac = black outline within the spinal canal, Spinal cord = Yellow shaded area within the spinal canal.
Figure 1. Bilsky grading for metastatic epidural spinal cord compression (MESCC) was demonstrated using axial CT, MRI (T2-weighted), and a picture for each grade (right to left). The red shaded region shows MESCC in each image. Grade 0/normal (Black outline): No metastatic epidural disease present. Low-grade MESCC (Blue outline); Grade 1a: Metastatic epidural soft tissue with no thecal sac deformity, or Grade 1b: Metastatic epidural soft tissue with thecal sac deformity but no spinal cord contact. High-grade MESCC (Red outline); Grade 1c: Metastatic epidural soft tissue touching the spinal cord with no discernible compression or displacement, or Grade 2: Metastatic epidural soft tissue cord compression with some surrounding cerebrospinal fluid still visible, or Grade 3: Metastatic epidural spinal cord compression without discernible surrounding cerebrospinal fluid. Thecal sac = black outline within the spinal canal, Spinal cord = Yellow shaded area within the spinal canal.
Cancers 14 04231 g001
Figure 2. Flow chart demonstrating the overall study design with inclusions and exclusions highlighted. MESCC = Metastatic epidural spinal cord compression.
Figure 2. Flow chart demonstrating the overall study design with inclusions and exclusions highlighted. MESCC = Metastatic epidural spinal cord compression.
Cancers 14 04231 g002
Figure 3. Subtle low-grade metastatic epidural disease at the left anterolateral spinal canal at T6. This was undercalled by all three radiologists and is challenging due to the paucity of bony change and location at the typical site for an epidural vein.
Figure 3. Subtle low-grade metastatic epidural disease at the left anterolateral spinal canal at T6. This was undercalled by all three radiologists and is challenging due to the paucity of bony change and location at the typical site for an epidural vein.
Cancers 14 04231 g003
Figure 4. Axial CT image of the cervicothoracic junction overcalled as low-grade epidural disease by two radiologists. No epidural disease was present on the corresponding MRI (not shown). Assessment is complex due to overlap and angulation of the lower cervical and upper thoracic spine in the axial plane with background degenerative changes.
Figure 4. Axial CT image of the cervicothoracic junction overcalled as low-grade epidural disease by two radiologists. No epidural disease was present on the corresponding MRI (not shown). Assessment is complex due to overlap and angulation of the lower cervical and upper thoracic spine in the axial plane with background degenerative changes.
Cancers 14 04231 g004
Table 1. CT scanner types and parameters.
Table 1. CT scanner types and parameters.
Parameter4-Slice64-Slice256-Slice384-Slice512-Slice
Pitch1.51.20.9840.80.531
Slice thickness (mm)55333
Collimation (mm)4 × 132 × 0.6128 × 0.625192 × 0.6256 × 0.625
kV120120100100100–120
Reference mAs180200250200200
Rotation time (s)0.50.50.50.50.5
Note- kV = Kilovoltage, mAs = Milliampere-seconds. All CT scanners were located at the National University Hospital, Singapore. Scanning was conducted with patients in a supine position in a craniocaudal direction. Intravenous contrast volume for all scans = 70–100 mL depending on the size of the patient (rate of 1.2–1.5 mL/s).
Table 2. MRI scanner types and parameters for axial T2-weighted images.
Table 2. MRI scanner types and parameters for axial T2-weighted images.
Parameter1.5-T1.5-T1.5-T3.0-T3.0-T
TR (msec)35003500400053005300
TE (msec)808090100100
Slice thickness (mm)55555
Gap (mm)66666
Field of view (mm2)200 × 200200 × 200160 × 160200 × 200160 × 160
Matrix512 × 512512 × 512320 × 320512 × 512640 × 640
Note- TR = repetition time, TE = echo time. All MRI scanners were located at the National University Hospital, Singapore. All scans were performed with patients in a supine position using a torso/body coil.
Table 3. Patient details and location of metastatic epidural spinal cord compression.
Table 3. Patient details and location of metastatic epidural spinal cord compression.
CharacteristicsPatients (Overall n = 101)
Age (years) *60 ± 11.6 (26–93)
Women47 (46.5)
Men54 (53.5)
Type of Cancer
Lung29 (28.7)
Breast21 (20.8)
Colorectal12 (11.9)
Renal cell carcinoma10 (9.9)
Prostate5 (5.0)
Hepatocellular carcinoma5 (5.0)
Multiple Myeloma4 (4.0)
Nasopharyngeal carcinoma3 (3.0)
Others12 (11.9)
Number of CT scans123
MESCC location per CT scan
Diffuse thoracic #32 (26.0)
C7-T27 (5.7)
T3-T1028 (22.8)
T11-L339 (31.7)
No epidural disease17 (13.8)
Note- MESCC = Metastatic epidural spinal cord compression. * Values presented as mean ± SD (range) for numerical and n (%) for categorical variables. # Two or more sites of MESCC.
Table 4. Reference standard gradings for metastatic epidural spinal cord compression.
Table 4. Reference standard gradings for metastatic epidural spinal cord compression.
Bilsky MESCC GradePer axial CT ImagePer CT Scan
Normal or 05642 (86.2)17 (13.8)
Low (1a or 1b)471 (7.2)16 (13.0)
High (1c or 2 or 3)432 (6.6)90 (73.2)
Total6545123
Note—Values are n (%) per CT image and CT scan (~50 images). MESCC = Metastatic Epidural Spinal Cord Compression.
Table 5. Radiologist interobserver variability for three-grade MESCC classification on CT.
Table 5. Radiologist interobserver variability for three-grade MESCC classification on CT.
Three-Grade MESCC (Normal, Low, and High)
Radiologist (per image)Kappa (95% CI)p-Value
PJ0.927 (0.920–0.934)<0.001
TYL0.928 (0.921–0.935)<0.001
YT0.931 (0.924–0.938)<0.001
Radiologist (per scan)
PJ0.692 (0.585–0.798)<0.001
TYL0.854 (0.777–0.931)<0.001
YT0.694 (0.587–0.801)<0.001
Table 6. Radiologist sensitivity, specificity, and AUCs for two-grade (normal/low versus high-grade) MESCC per CT image.
Table 6. Radiologist sensitivity, specificity, and AUCs for two-grade (normal/low versus high-grade) MESCC per CT image.
RadiologistKappa (95% CI)p-ValueSensitivity (95% CI)Specificity (95% CI)AUC (95% CI)
PJ0.967 (0.962–0.972)<0.00156.48 (51.66–61.21)99.90 (99.79–99.96)0.782 (0.752–0.812)
TYL0.978 (0.974–0.982)<0.00177.78 (73.56–81.61)99.49 (99.28–99.66)0.886 (0.863–0.910)
YT0.973 (0.969–0.977)<0.00165.74 (61.05–70.21)99.85 (99.72–99.93)0.828 (0.800–0.856)
Table 7. Radiologist sensitivity, specificity, and AUCs for two-grade (normal/low versus high-grade) MESCC per CT scan.
Table 7. Radiologist sensitivity, specificity, and AUCs for two-grade (normal/low versus high-grade) MESCC per CT scan.
RadiologistKappa (95% CI)p-ValueSensitivity (95% CI)Specificity (95% CI)AUC (95% CI)
PJ0.653 (0.516–0.789)<0.00174.44 (64.16–83.06)100.00 (89.42–100.00)0.872 (0.827–0.918)
TYL0.905 (0.833–0.976)<0.00194.44 (87.51–98.17)93.94 (79.77–99.26)0.942 (0.894–0.990)
YT0.688 (0.558–0.819)<0.00177.78 (67.79–85.87)96.97 (84.24–99.92)0.874 (0.821–0.926)
Original reporting radiologist0.213 (0.036–0.391)0.01948.89 (38.20–59.65)90.91 (75.67–98.08)0.699 (0.627–0.771)
Table 8. Radiologist sensitivity, specificity, and AUCs for two-grade (normal versus low/high-grade) MESCC per CT image.
Table 8. Radiologist sensitivity, specificity, and AUCs for two-grade (normal versus low/high-grade) MESCC per CT image.
RadiologistKappa (95% CI)p-ValueSensitivity (95% CI)Specificity (95% CI)AUC (95% CI)
PJ0.949 (0.943–0.955)<0.00177.30 (74.42–79.99)99.03 (98.73–99.26)0.882 (0.865–0.898)
TYL0.936 (0.929–0.943)<0.00173.31 (70.30–76.17)98.49 (98.14–98.79)0.859 (0.842–0.877)
YT0.948 (0.941–0.954)<0.00186.82 (84.44–88.96)97.50 (97.06–97.89)0.922 (0.908–0.935)
Table 9. Radiologist sensitivity, specificity, and AUCs for two-grade (normal versus low/high-grade) MESCC per CT scan.
Table 9. Radiologist sensitivity, specificity, and AUCs for two-grade (normal versus low/high-grade) MESCC per CT scan.
RadiologistKappa (95% CI)p-ValueSensitivity (95% CI)Specificity (95% CI)AUC (95% CI)
PJ0.947 (0.899–0.995)<0.00198.11 (93.35–99.77)82.35 (56.57–96.20)0.902 (0.808–0.997)
TYL0.917 (0.858–0.977)<0.00198.11 (93.35–99.77)64.71 (38.33–85.79)0.814 (0.696–0.932)
YT0.866 (0.787–0.945)<0.00191.51 (84.49–96.04)82.35 (56.57–96.20)0.869 (0.772–0.966)
Original reporting radiologist0.095 (−0.098–0.287)0.33344.34 (34.69–54.31)100.00 (80.49–100.00)0.722 (0.674–0.769)
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Hallinan, J.T.P.D.; Ge, S.; Zhu, L.; Zhang, W.; Lim, Y.T.; Thian, Y.L.; Jagmohan, P.; Kuah, T.; Lim, D.S.W.; Low, X.Z.; et al. Diagnostic Accuracy of CT for Metastatic Epidural Spinal Cord Compression. Cancers 2022, 14, 4231. https://doi.org/10.3390/cancers14174231

AMA Style

Hallinan JTPD, Ge S, Zhu L, Zhang W, Lim YT, Thian YL, Jagmohan P, Kuah T, Lim DSW, Low XZ, et al. Diagnostic Accuracy of CT for Metastatic Epidural Spinal Cord Compression. Cancers. 2022; 14(17):4231. https://doi.org/10.3390/cancers14174231

Chicago/Turabian Style

Hallinan, James Thomas Patrick Decourcy, Shuliang Ge, Lei Zhu, Wenqiao Zhang, Yi Ting Lim, Yee Liang Thian, Pooja Jagmohan, Tricia Kuah, Desmond Shi Wei Lim, Xi Zhen Low, and et al. 2022. "Diagnostic Accuracy of CT for Metastatic Epidural Spinal Cord Compression" Cancers 14, no. 17: 4231. https://doi.org/10.3390/cancers14174231

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop