Patients’ and observers’ characteristics
Since April 2011, ten consecutive patients who were irradiated to the pelvic region were enrolled in this multiple-observer contouring study. The ethical committee of Hokkaido University Hospital approved this study (number 010-0305). Patients who had had prostatectomy were excluded. The individual patients’ characteristics are listed in Table
1. Of the five patients with bladder tumors, two patients received ureteral stents prior to radiotherapy. Fiducial markers were not placed in any of the patients.
Table 1
Patient characteristics
A | 90 | Female | Bladder |
B | 70 | Female | Uterus |
C | 83 | Female | Bladder |
D | 71 | Female | Bladder |
E | 68 | Male | Prostate |
F | 83 | Male | Bladder |
G | 90 | Male | Bladder |
H | 77 | Male | Prostate |
I | 69 | Male | Prostate |
J | 74 | Male | Prostate |
Five physicians (four experienced radiation oncologists and one senior resident of the Department of Radiation Oncology who had worked in genito-urinary service) were recruited for the study (KN, RK, TI, SO, KY, and KH). The clinical experience of radiotherapy of all observers was ranged from 3 to 8 years with an average experience of 5.6 years.
CBCT image acquisition
Patients with bladder cancer were asked to void just before their treatment during the treatment course, and no other bowel or bladder preparation protocol including diet-related instruction was offered to any of the 10 patients. All CBCT datasets were acquired weekly in the supine position, immediately after initial setup to skin marks. CBCT images were not used to adjust the patient’s position in this study period.
All patients were imaged and treated on a Varian Clinac iX Linear Accelerator (Varian Medical Systems, Palo Alto, CA, USA) using the kV imaging system. The CBCT images were acquired using standard factory settings of 125 kVp, 80 mA, and 20 ms per projection with a half bow-tie filter. Images were reconstructed at an axial slice thickness of 0.25 cm.
Contouring protocol
For delineation of the organ boundaries, we used the first CBCT dataset of each patient that contained the entire bladder and prostate during the treatment course.
All observers were asked to delineate the outer contour of the whole bladder and prostate without margin for microscopic extension and seminal vesicles. In all cases the bladder was contoured as a solid organ. Contouring was performed in a blinded fashion, i.e., each observer could use only one image dataset of the patient at the time of delineation. Access to the structures drawn by other participants or the other imaging modalities (e.g., treatment planning CT, diagnostic CT, or MRI) as well as the help of a radiologist was not permitted. Contouring was carried out in the treatment planning system (Eclipse ver. 8.9, Varian Medical Systems, Inc.) using the standard tools available. Observers were free to modify window range and level of the images as preferred, and interpolation of the contours between slices was allowed. Intra-observer error was not investigated as part of this study.
Inter-observer variation analysis
The total encompassing delineated volume and the overlapping volume between the observers’ contours were calculated using the Eclipse planning system Boolean function.
To assess inter-observer variations in organ volumes, we calculated coefficients of variation (COV = standard deviation/mean volume) for the bladder and prostate. The COVs of all observers’ contours per patient were calculated and averaged over all patients.
To evaluate the inter-observer concordance, the generalized conformity index (CI
gen), defined as the ratio of the sum of all overlapping volumes between pairs of observers and the sum of all overlapping and all non-overlapping volumes between the same pairs [
18], was used, as follows:
A CIgen of 1 indicates 100% concordance for the volume segmentation, a CIgen of 0.5 indicates 50% agreement between observers for the encompassing volume, a CIgen of 0 indicates no concordance in delineation. The CIgens were calculated per patient and averaged over all patients.
Coordinates of the center-of-mass (COM) of each structure in 3D were also extracted. COM displacement values along the left-right (LR), anterior-posterior (AP), and cranial-caudal (CC) direction were analyzed. As the overall mean of standard deviation, the root mean square (RMS) of the total COM standard deviation (σ) on CBCT was calculated, as follows:
where σ
i
indicates the standard deviation of the COM displacement value of the structure in patient i drawn by the respective observer in a given direction.
To evaluate the reliability of this study, we calculated the intra-class correlation coefficients (ICC(2,
k)), where
k represents the number of observers. The ICC is a tool for reliability analysis, which is defined from the variance components as
where the subscripts
ws and
bs denote within-subject and between-subjects variance, respectively. As the true value of the variance is unknown, we use estimates from analysis of variance (ANOVA) analysis, which provides the variance components with respective mean squares between patient cases (MS
bpat), within one patient case (MS
wpat), between observers (MS
obs), and between error terms (MS
err). As different forms of ICC are described in the literature, we selected ICC(2,k) for the situation in which some physicians (observers) of the department delineated organ boundaries in multiple patients, once for each patient. The ICC can be used to assess the overall reliability of
k observers in contouring all
n given cases (ICC(2,
k)), as follows:
ICC values < 0.4 indicate poor reliability, ICC values between 0.4 and 0.6 indicate moderate reliability, and ICC values > 0.6 or 0.8 denote substantial or excellent reliability, respectively [
19].
Statistical analysis was performed with JMP 9.0.3 (SAS Institute, Cary, NC, USA) and SPSS 11.5 (SPSS Inc., Chicago IL). Statistical significance of the outcome was assumed for p<0.05.