Uncertainties in volume delineation in radiation oncology: A systematic review and recommendations for future studies

doi:10.1016/j.radonc.2016.09.009

Radiotherapy and Oncology

Volume 121, Issue 2, November 2016, Pages 169-179

https://doi.org/10.1016/j.radonc.2016.09.009 Get rights and content

Abstract

Background and purpose

Volume delineation is a well-recognised potential source of error in radiotherapy. Whilst it is important to quantify the degree of interobserver variability (IOV) in volume delineation, the resulting impact on dosimetry and clinical outcomes is a more relevant endpoint. We performed a literature review of studies evaluating IOV in target volume and organ-at-risk (OAR) delineation in order to analyse these with respect to the metrics used, reporting of dosimetric consequences, and use of statistical tests.

Methods and materials

Medline and Pubmed databases were queried for relevant articles using keywords. We included studies published in English between 2000 and 2014 with more than two observers.

Results

119 studies were identified covering all major tumour sites. CTV (n = 47) and GTV (n = 38) were most commonly contoured. Median number of participants and data sets were 7 (3–50) and 9 (1–132) respectively. There was considerable heterogeneity in the use of metrics and methods of analysis. Statistical analysis of results was reported in 68% (n = 81) and dosimetric consequences in 21% (n = 25) of studies.

Conclusion

There is a lack of consistency in conducting and reporting analyses from IOV studies. We suggest a framework to use for future studies evaluating IOV.

Section snippets

Methods

The Medline and Pubmed databases were queried for relevant articles using the keywords “radiotherapy” and “volume delineation”, “contouring”, “observer variation”, “interobserver variability”, “variation”, “systematic error”, “quality assurance”, “delineation”, “interobserver” and “intraobserver” to identify articles which evaluated interobserver variability in target or organ-at-risk (OAR) volume delineation. Studies had to fulfil the following criteria to be selected for this review:

•
Multiple

Studies evaluating IOV in volume delineation

119 studies were identified for the following clinical sites: breast [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], bladder [28], [29], prostate [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], lung [45], [46], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [61], oesophagus [62], [63], stomach [64], [65], pancreas [66], [67], liver [68], rectum [69]

Discussion

Volume delineation in radiotherapy is one of the earliest steps in the planning process and as such accuracy is crucial as otherwise this leads to a systematic error downstream. Comparison with pathology remains the gold standard when measuring accuracy, however this is rarely achievable [134], [135]. A surrogate endpoint is to minimise variation in volume delineation by multiple observers, presuming that the volume common to all is closest to the ground truth.

In a review of clinical

Conclusions

This review has highlighted the lack of consistency in conducting and reporting analyses from IOV studies making comparisons difficult. We suggest a set of reporting standards for use in future studies of IOV in volume delineation.

Funding

This project was funded by NHMRC project grant number 1102198.

Conflict of interest

The authors have no conflict of interest in relation to this manuscript.

Acknowledgements

We thank Dr Wei Xuan for his help with statistical interpretation of studies.

References (156)

M.G. Jameson et al.
Correlation of contouring variation with modeled outcome for conformal non-small cell lung cancer radiotherapy
Radiother Oncol
(2014)
D.C. Weber et al.
QA makes a clinical trial stronger: evidence-based medicine in radiation therapy
Radiother Oncol
(2012)
R.A. Abrams et al.
Failure to adhere to protocol specified radiation therapy guidelines was associated with decreased survival in RTOG 9704–a phase III trial of adjuvant chemotherapy and chemoradiotherapy for patients with resected adenocarcinoma of the pancreas
Int J Radiat Oncol Biol Phys
(2012)
C.W. Hurkmans et al.
Variability in target volume delineation on CT scans of the breast
Int J Radiat Oncol Biol Phys
(2001)
H. Struikmans et al.
Interobserver variability of clinical target volume delineation of glandular breast tissue and of boost volume in tangential breast irradiation
Radiother Oncol
(2005)
E.K. Wong et al.
Consistency in seroma contouring for partial breast radiotherapy: impact of guidelines
Int J Radiat Oncol Biol Phys
(2006)
D.M. Landis et al.
Variability among breast radiation oncologists in delineation of the postsurgical lumpectomy cavity
Int J Radiat Oncol Biol Phys
(2007)
R.P. Petersen et al.
Target volume delineation for partial breast radiotherapy planning: clinical characteristics associated with low interobserver concordance
Int J Radiat Oncol Biol Phys
(2007)
X.A. Li et al.
Variability of target and normal structure delineation for breast cancer radiotherapy: an RTOG Multi-Institutional and Multiobserver Study
Int J Radiat Oncol Biol Phys
(2009)
V.K. Reed et al.
Automatic segmentation of whole breast using atlas approach and deformable image registration
Int J Radiat Oncol Biol Phys
(2009)

T. Shaikh et al.

Improvement in interobserver accuracy in delineation of the lumpectomy cavity using fiducial markers

Int J Radiat Oncol Biol Phys

(2010)

A.M. van Mourik et al.

Multiinstitutional study on target volume delineation variation in breast radiotherapy in the presence of guidelines

Radiother Oncol

(2010)

V. Batumalai et al.

Interobserver variability in clinical target volume delineation in tangential breast irradiation: a comparison between radiation oncologists and radiation therapists

Clinic Oncol (R Coll Radiol)

(2011)

M. Giezen et al.

Magnetic resonance imaging- versus computed tomography-based target volume delineation of the glandular breast tissue (clinical target volume breast) in breast-conserving therapy: an exploratory study

Int J Radiat Oncol Biol Phys

(2011)

M. Jolicoeur et al.

Localization of the surgical bed using supine magnetic resonance and computed tomography scan fusion for planification of breast interstitial brachytherapy

Radiother Oncol

(2011)

L.J. Boersma et al.

Reducing interobserver variation of boost-CTV delineation in breast conserving radiation therapy using a pre-operative CT and delineation guidelines

Radiother Oncol

(2012)

R. Kosztyla et al.

Evaluation of dosimetric consequences of seroma contour variability in accelerated partial breast irradiation using a constructed representative seroma contour

Int J Radiat Oncol Biol Phys

(2012)

G. Lee et al.

Evaluation in variability in seroma delineation between clinical specialist radiation therapist and radiation oncologist for adjuvant breast irradiation

Pract Radiat Oncol

(2012)

F. van der Leij et al.

Target volume delineation in external beam partial breast irradiation: less inter-observer variation with preoperative- compared to postoperative delineation

Radiother Oncol

(2014)

J. Yang et al.

Statistical modeling approach to quantitative analysis of interobserver variability in breast contouring

Int J Radiat Oncol Biol Phys

(2014)

G.J. Meijer et al.

Three-dimensional analysis of delineation errors, setup errors, and organ motion during radiotherapy of bladder cancer

Int J Radiat Oncol Biol Phys

(2003)

B. Seddon et al.

Target volume definition in conformal radiotherapy for prostate cancer: quality assurance in the MRC RT-01 trial

Radiother Oncol

(2000)

J.E. Livsey et al.

Do differences in target volume definition in prostate cancer lead to clinically relevant differences in normal tissue toxicity?

Int J Radiat Oncol Biol Phys

(2004)

C.A. Lawton et al.

Variation in the definition of clinical target volumes for pelvic nodal conformal radiation therapy for prostate cancer

Int J Radiat Oncol Biol Phys

(2009)

D.M. Mitchell et al.

Assessing the effect of a contouring protocol on postprostatectomy radiotherapy clinical target volumes and interphysician variation

Int J Radiat Oncol Biol Phys

(2009)

E. Szumacher et al.

Effectiveness of educational intervention on the congruence of prostate and rectal contouring as compared with a gold standard in three-dimensional radiotherapy for prostate

Int J Radiat Oncol Biol Phys

(2010)

P. Ost et al.

Delineation of the postprostatectomy prostate bed using computed tomography: interobserver variability following the EORTC delineation guidelines

Int J Radiat Oncol Biol Phys

(2011)

E.L. Khoo et al.

Prostate contouring variation: can it be fixed?

Int J Radiat Oncol Biol Phys

(2012)

R. Moeckli et al.

Physical considerations on discrepancies in target volume delineation

Z Med Phys

(2009)

C.B. Caldwell et al.

Observer variation in contouring gross tumor volume in patients with poorly defined non-small-cell lung tumors on CT: the impact of 18FDG-hybrid PET fusion

Int J Radiat Oncol Biol Phys

(2001)

P. Bowden et al.

Measurement of lung tumor volumes using three-dimensional computer planning software

Int J Radiat Oncol Biol Phys

(2002)

P. Giraud et al.

Conformal radiotherapy for lung cancer: different delineation of the gross tumor volume (GTV) by radiologists and radiation oncologists

Radiother Oncol

(2002)

J. Van de Steene et al.

Definition of gross tumor volume in lung cancer: inter-observer variability

Radiother Oncol

(2002)

R.J. Steenbakkers et al.

Reduction of observer variation using matched CT-PET for lung cancer delineation: a three-dimensional analysis

Int J Radiat Oncol Biol Phys

(2006)

L. Kepka et al.

Delineation variation of lymph node stations for treatment planning in lung cancer radiotherapy

Radiother Oncol

(2007)

A. van Baardwijk et al.

PET-CT-based auto-contouring in non-small-cell lung cancer correlates with pathology and reduces interobserver variability in the delineation of the primary tumor and involved nodal volumes

Int J Radiat Oncol Biol Phys

(2007)

G.G. Hanna et al.

18F-fluorodeoxyglucose positron emission tomography/computed tomography-based radiotherapy target volume definition in non-small-cell lung cancer: delineation by radiation oncologists vs. joint outlining with a PET radiologist?

Int J Radiat Oncol Biol Phys

(2010)

A.V. Louie et al.

Inter-observer and intra-observer reliability for lung cancer target volume delineation in the 4D-CT era

Radiother Oncol

(2010)

F.O. Spoelstra et al.

Variations in target volume definition for postoperative radiotherapy in stage III non-small-cell lung cancer: analysis of an international contouring study

Int J Radiat Oncol Biol Phys

(2010)

G. Altorjai et al.

Cone-beam CT-based delineation of stereotactic lung targets: the influence of image modality and target size on interobserver variability

Int J Radiat Oncol Biol Phys

(2012)

H. Vorwerk et al.

The delineation of target volumes for radiotherapy of lung cancer patients

Radiother Oncol

(2009)

P. Tai et al.

Improving the consistency in cervical esophageal target volume definition by special training

Int J Radiat Oncol Biol Phys

(2002)

S. Gwynne et al.

Toward semi-automated assessment of target volume delineation in radiotherapy trials: the SCOPE 1 pretrial test case

Int J Radiat Oncol Biol Phys

(2012)

E.P. Jansen et al.

Interobserver variation of clinical target volume delineation in gastric cancer

Int J Radiat Oncol Biol Phys

(2010)

N.K.G. Jensen et al.

Dynamic contrast enhanced CT aiding gross tumor volume delineation of liver tumors: An interobserver variability study

Radiother Oncol

(2014)

C.D. Fuller et al.

Prospective randomized double-blind pilot study of site-specific consensus atlas implementation for rectal cancer target volume delineation in the cooperative group setting

Int J Radiat Oncol Biol Phys

(2011)

P. Mavroidis et al.

Consequences of anorectal cancer atlas implementation in the cooperative group setting: radiobiologic analysis of a prospective randomized in silico target delineation study

Radiother Oncol

(2014)

J. Buijsen et al.

FDG-PET-CT reduces the interobserver variability in rectal tumor delineation

Radiother Oncol

(2012)

J. Nijkamp et al.

Target volume delineation variation in radiotherapy for early stage rectal cancer in the Netherlands

Radiother Oncol

(2012)

J.T. Whaley et al.

Clinical utility of integrated positron emission tomography/computed tomography imaging in the clinical management and radiation treatment planning of locally advanced rectal cancer

Pract Radiat Oncol

(2014)

Cited by (249)

Not without Context—A Multiple Methods Study on Evaluation and Correction of Automated Brain Tumor Segmentations by Experts
2024, Academic Radiology
Brain tumor segmentations are integral to the clinical management of patients with glioblastoma, the deadliest primary brain tumor in adults. The manual delineation of tumors is time-consuming and highly provider-dependent. These two problems must be addressed by introducing automated, deep-learning-based segmentation tools. This study aimed to identify criteria experts use to evaluate the quality of automatically generated segmentations and their thought processes as they correct them.
Multiple methods were used to develop a detailed understanding of the complex factors that shape experts’ perception of segmentation quality and their thought processes in correcting proposed segmentations. Data from a questionnaire and semistructured interview with neuro-oncologists and neuroradiologists were collected between August and December 2021 and analyzed using a combined deductive and inductive approach.
Brain tumors are highly complex and ambiguous segmentation targets. Therefore, physicians rely heavily on the given context related to the patient and clinical context in evaluating the quality and need to correct brain tumor segmentation. Most importantly, the intended clinical application determines the segmentation quality criteria and editing decisions. Physicians’ personal beliefs and preferences about the capabilities of AI algorithms and whether questionable areas should not be included are additional criteria influencing the perception of segmentation quality and appearance of an edited segmentation.
Our findings on experts’ perceptions of segmentation quality will allow the design of improved frameworks for expert-centered evaluation of brain tumor segmentation models. In particular, the knowledge presented here can inspire the development of brain tumor-specific metrics for segmentation model training and evaluation.
Radiotherapy trial quality assurance processes: a systematic review
2024, The Lancet Oncology
Quality assurance remains a neglected component of many trials, particularly for technical interventions, such as surgery and radiotherapy, for which quality of treatment is an important component in defining outcomes. We aimed to evaluate evidence for the processes used in radiotherapy quality assurance of clinical trials. A systematic review was undertaken focusing on use of a pre-trial outlining benchmark case and subsequent on-trial individual case reviews of outlining for recruited patients. These pre-trial and on-trial checks are used to ensure consistency and standardisation of treatment for each patient recruited to the trial by confirming protocol compliance. Non-adherence to the trial protocol has been shown to have a negative effect on trial outcomes. 29 studies published between January, 2000, and December, 2022, were identified that reported on either outlining benchmark case results or outlining individual case review results, or both. The trials identified varied in their use of radiotherapy quality assurance practices and reporting of outcomes was inconsistent. Deviations from trial protocols were frequent, particularly regarding outlining. Studies correlating benchmark case results with on-trial individual case reviews provided mixed results, meaning firm conclusions could not be drawn regarding the influence of the pre-trial benchmark case on subsequent on-trial performance. The optimal radiotherapy quality assurance processes were unclear, and there was little evidence available. Improved reporting of outcomes from radiotherapy quality assurance programmes is needed to develop an evidence base for the optimal approach to radiotherapy quality assurance in trials.
Development of Prostate Bed Delineation Consensus Guidelines for Magnetic Resonance Image-Guided Radiotherapy and Assessment of Its Effect on Interobserver Variability
2024, International Journal of Radiation Oncology Biology Physics
The use of magnetic resonance imaging (MRI) in radiotherapy planning is becoming more widespread, particularly with the emergence of MRI-guided radiotherapy systems. Existing guidelines for defining the prostate bed clinical target volume (CTV) show considerable heterogeneity. This study aimed to establish baseline interobserver variability (IOV) for prostate bed CTV contouring on MRI, develop international consensus guidelines, and evaluate its effect on IOV.
Participants delineated the CTV on 3 MRI scans, obtained from the Elekta Unity MR-Linac, as per their normal practice. Radiation oncologist contours were visually examined for discrepancies, and interobserver comparisons were evaluated against simultaneous truth and performance level estimation (STAPLE) contours using overlap metrics (Dice similarity coefficient and Cohen's kappa), distance metrics (mean distance to agreement and Hausdorff distance), and volume measurements. A literature review of postradical prostatectomy local recurrence patterns was performed and presented alongside IOV results to the participants. Consensus guidelines were collectively constructed, and IOV assessment was repeated using these guidelines.
Sixteen radiation oncologists’ contours were included in the final analysis. Visual evaluation demonstrated significant differences in the superior, inferior, and anterior borders. Baseline IOV assessment indicated moderate agreement for the overlap metrics while volume and distance metrics demonstrated greater variability. Consensus for optimal prostate bed CTV boundaries was established during a virtual meeting. After guideline development, a decrease in IOV was observed. The maximum volume ratio decreased from 4.7 to 3.1 and volume coefficient of variation reduced from 40% to 34%. The mean Dice similarity coefficient rose from 0.72 to 0.75 and the mean distance to agreement decreased from 3.63 to 2.95 mm.
Interobserver variability in prostate bed contouring exists among international genitourinary experts, although this is lower than previously reported. Consensus guidelines for MRI-based prostate bed contouring have been developed, and this has resulted in an improvement in contouring concordance. However, IOV persists and strategies such as an education program, development of a contouring atlas, and further refinement of the guidelines may lead to additional improvements.
Assessment of deep learning-based auto-contouring on interobserver consistency in target volume and organs-at-risk delineation for breast cancer: Implications for RTQA program in a multi-institutional study
2024, Breast
To quantify interobserver variation (IOV) in target volume and organs-at-risk (OAR) contouring across 31 institutions in breast cancer cases and to explore the clinical utility of deep learning (DL)-based auto-contouring in reducing potential IOV.
In phase 1, two breast cancer cases were randomly selected and distributed to multiple institutions for contouring six clinical target volumes (CTVs) and eight OAR. In Phase 2, auto-contour sets were generated using a previously published DL Breast segmentation model and were made available for all participants. The difference in IOV of submitted contours in phases 1 and 2 was investigated quantitatively using the Dice similarity coefficient (DSC) and Hausdorff distance (HD). The qualitative analysis involved using contour heat maps to visualize the extent and location of these variations and the required modification.
Over 800 pairwise comparisons were analysed for each structure in each case. Quantitative phase 2 metrics showed significant improvement in the mean DSC (from 0.69 to 0.77) and HD (from 34.9 to 17.9 mm). Quantitative analysis showed increased interobserver agreement in phase 2, specifically for CTV structures (5–19 %), leading to fewer manual adjustments. Underlying IOV differences causes were reported using a questionnaire and hierarchical clustering analysis based on the volume of CTVs.
DL-based auto-contours improved the contour agreement for OARs and CTVs significantly, both qualitatively and quantitatively, suggesting its potential role in minimizing radiation therapy protocol deviation.
Random Forest Modeling of Acute Toxicity in Anal Cancer: Effects of Peritoneal Cavity Contouring Approaches on Model Performance
2024, International Journal of Radiation Oncology Biology Physics
Our purpose was to analyze the effect on gastrointestinal (GI) toxicity models when their dose-volume metrics predictors are derived from segmentations of the peritoneal cavity after different contouring approaches.
A random forest machine learning approach was used to predict acute grade ≥3 GI toxicity from dose-volume metrics and clinicopathologic factors for 246 patients (toxicity incidence = 9.5%) treated with definitive chemoradiation for squamous cell carcinoma of the anus. Three types of random forest models were constructed based on different bowel bag segmentation approaches: (1) physician-delineated after Radiation Therapy Oncology Group (RTOG) guidelines, (2) autosegmented by a deep learning model (nnU-Net) following RTOG guidelines, and (3) autosegmented but spanning the entire bowel space. Each model type was evaluated using repeated cross-validation (100 iterations; 50%/50% training/test split). The performance of the models was assessed using area under the precision-recall curve (AUPRC) and the receiver operating characteristic curve (AUROCC), as well as optimal F1 score.
When following RTOG guidelines, the models based on the nnU-Net auto segmentations (mean values: AUROCC, 0.71 ± 0.07; AUPRC, 0.42 ± 0.09; F1 score, 0.46 ± 0.08) significantly outperformed (P < .001) those based on the physician-delineated contours (mean values: AUROCC, 0.67 ± 0.07; AUPRC, 0.34 ± 0.08; F1 score, 0.36 ± 0.07). When spanning the entire bowel space, the performance of the autosegmentation models improved considerably (mean values: AUROCC, 0.87 ± 0.05; AUPRC, 0.70 ± 0.09; F1 score, 0.68 ± 0.09).
Random forest models were superior at predicting acute grade ≥3 GI toxicity when based on RTOG-defined bowel bag autosegmentations rather than physician-delineated contours. Models based on autosegmentations spanning the entire bowel space show further considerable improvement in model performance. The results of this study should be further validated using an external data set.
Experience of Implementing Deep Learning–Based Automatic Contouring in Breast Radiation Therapy Planning: Insights From Over 2000 Cases
2024, International Journal of Radiation Oncology Biology Physics
This study evaluated the impact and clinical utility of an auto-contouring system for radiation therapy treatments.
The auto-contouring system was implemented in 2019. We evaluated data from 2428 patients who underwent adjuvant breast radiation therapy before and after the system's introduction. We collected the treatment's finalized contours, which were reviewed and revised by a multidisciplinary team. After implementation, the treatment contours underwent a finalization process that involved manual review and adjustment of the initial auto-contours. For the preimplementation group (n = 369), auto-contours were generated retrospectively. We compared the auto-contours and final contours using the Dice similarity coefficient (DSC) and the 95% Hausdorff distance (HD95).
We analyzed 22,215 structures from final and corresponding auto-contours. The final contours were generally larger, encompassing more slices in the superior or inferior directions. Among organs at risk (OAR), the heart, esophagus, spinal cord, and contralateral breast demonstrated significantly increased DSC and decreased HD95 postimplementation (all P < .05), except for the lungs, which presented inaccurate segmentation. Among target volumes, CTVn_L2, L3, L4, and the internal mammary node showed increased DSC and decreased HD95 postimplementation (all P < .05), although the increase was less pronounced than the OAR outcomes. The analysis also covered factors contributing to significant differences, pattern identification, and outlier detection.
In our study, the adoption of an auto-contouring system was associated with an increased reliance on automated settings, underscoring its utility and the potential risk of automation bias. Given these findings, we underscore the importance of considering the integration of stringent risk assessments and quality management strategies as a precautionary measure for the optimal use of such systems.

View all citing articles on Scopus

View full text

Systematic reviewUncertainties in volume delineation in radiation oncology: A systematic review and recommendations for future studies

Abstract

Background and purpose

Methods and materials

Results

Conclusion

Section snippets

Methods

Studies evaluating IOV in volume delineation

Discussion

Conclusions

Funding

Conflict of interest

Acknowledgements

Radiother Oncol

Radiother Oncol

Int J Radiat Oncol Biol Phys

Int J Radiat Oncol Biol Phys

Radiother Oncol

Int J Radiat Oncol Biol Phys

Int J Radiat Oncol Biol Phys

Int J Radiat Oncol Biol Phys

Int J Radiat Oncol Biol Phys

Int J Radiat Oncol Biol Phys

Int J Radiat Oncol Biol Phys

Radiother Oncol

Clinic Oncol (R Coll Radiol)

Int J Radiat Oncol Biol Phys

Radiother Oncol

Radiother Oncol

Int J Radiat Oncol Biol Phys

Pract Radiat Oncol

Radiother Oncol

Int J Radiat Oncol Biol Phys

Int J Radiat Oncol Biol Phys

Radiother Oncol

Int J Radiat Oncol Biol Phys

Int J Radiat Oncol Biol Phys

Int J Radiat Oncol Biol Phys

Int J Radiat Oncol Biol Phys

Int J Radiat Oncol Biol Phys

Int J Radiat Oncol Biol Phys

Z Med Phys

Int J Radiat Oncol Biol Phys

Int J Radiat Oncol Biol Phys

Radiother Oncol

Radiother Oncol

Int J Radiat Oncol Biol Phys

Radiother Oncol

Int J Radiat Oncol Biol Phys

Int J Radiat Oncol Biol Phys

Radiother Oncol

Int J Radiat Oncol Biol Phys

Int J Radiat Oncol Biol Phys

Radiother Oncol

Int J Radiat Oncol Biol Phys

Int J Radiat Oncol Biol Phys

Int J Radiat Oncol Biol Phys

Radiother Oncol

Int J Radiat Oncol Biol Phys

Radiother Oncol

Radiother Oncol

Radiother Oncol

Pract Radiat Oncol

Systematic review
Uncertainties in volume delineation in radiation oncology: A systematic review and recommendations for future studies