Effect of Dosimetric Outliers on the Performance of a Commercial Knowledge-Based Planning Solution

doi:10.1016/j.ijrobp.2015.11.011

International Journal of Radiation OncologyBiologyPhysics

Volume 94, Issue 3, 1 March 2016, Pages 469-477

https://doi.org/10.1016/j.ijrobp.2015.11.011 Get rights and content

Purpose

RapidPlan, a commercial knowledge-based planning solution, uses a model library containing the geometry and associated dosimetry of existing plans. This model predicts achievable dosimetry for prospective patients that can be used to guide plan optimization. However, it is unknown how suboptimal model plans (outliers) influence the predictions or resulting plans. We investigated the effect of, first, removing outliers from the model (cleaning it) and subsequently adding deliberate dosimetric outliers.

Methods and Materials

Clinical plans from 70 head and neck cancer patients comprised the uncleaned (UC) Model_UC, from which outliers were cleaned (C) to create Model_C. The last 5 to 40 patients of Model_C were replanned with no attempt to spare the salivary glands. These substantial dosimetric outliers were reintroduced to the model in increments of 5, creating Model₅ to Model₄₀ (Model_5-40). These models were used to create plans for a 10-patient evaluation group. Plans from Model_UC and Model_C, and Model_C and Model_5-40 were compared on the basis of boost (B) and elective (E) target volume homogeneity indexes (HI_B/HI_E) and mean doses to oral cavity, composite salivary glands (comp_sal) and swallowing (comp_swal) structures.

Results

On average, outlier removal (Model_C vs Model_UC) had minimal effects on HI_B/HI_E (0%-0.4%) and sparing of organs at risk (mean dose difference to oral cavity and comp_sal/comp_swal were ≤0.4 Gy). Model_5-10 marginally improved comp_sal sparing, whereas adding a larger number of outliers (Model_20-40) led to deteriorations in comp_sal up to 3.9 Gy, on average. These increases are modest compared to the 14.9 Gy dose increases in the added outlier plans, due to the placement of optimization objectives below the inferior boundary of the dose-volume histogram-predicted range.

Conclusions

Overall, dosimetric outlier removal from or addition of 5 to 10 outliers to a 70-patient model had marginal effects on resulting plan quality. Although the addition of >20 outliers deteriorated plan quality, the effect was modest. In this study, RapidPlan demonstrated robustness for moderate proportions of salivary gland dosimetric outliers.

Introduction

Automated solutions to inverse radiation therapy treatment planning are being developed to minimize the variability in plan quality associated with manual planning and to increase planning efficiency 1, 2. RapidPlan (Varian Medical Systems, Palo Alto, CA) is a commercial knowledge-based planning solution 3, 4, 5, 6, 7, 8, 9, 10, 11 derived from earlier work 12, 13, which uses a model based on a library of previous plans. Regression analysis is used to discern correlations between the geometric and dosimetric features of the planning target volumes (PTVs) and organs at risk (OARs) of the library plans. The model can be used to predict a range of achievable OAR dose-volume histograms (DVHs) for new patients. RapidPlan (RP) subsequently guides intensity modulated radiation therapy or RP volumetric modulated arc therapy optimization process in the Eclipse treatment planning system by placing a line of optimization objectives along the inferior boundary of the DVH prediction range. The broadness of the prediction range and, hence, the optimization objective placement is influenced by the goodness of fit between the geometric and dosimetric attributes of the model, along with the similarity of a prospective patient to the model population. Although knowledge-based planning using RP is in its infancy, promising results have been reported for different treatment sites 14, 15, 16.

RP performs statistical analysis of all modeled structures, thereby allowing identification of OARs which deviate from the bulk of the model population. Because of concerns that such “outliers” reduce the goodness of fit between geometry and dosimetry (9) and could, in turn, negatively influence model performance, removal or replanning of outlier OARs is recommended by the vendor (17). This process, however, is time consuming and subjective and must be repeated for each newly created or modified model. It is therefore useful to investigate the impact of this step on the created models. Although previous work illustrated the importance of having a broad range of OAR geometries in a model to accommodate prospective patients with varying geometries (16), the effect of dosimetric outliers in the model library on RP performance has not been systematically investigated.

The present study investigated the effect of dosimetric outliers on the resulting quality of RP plans generated for complex head and neck cancer (HNC) patients. Dosimetric and geometric outliers were removed from the model library, and the effect on plan quality of deliberately contaminating the model with plans in which the salivary glands were not actively spared was investigated.

Section snippets

Methods and Materials

All created models were populated with HNC RapidArc plans, planned with a simultaneous integrated boost technique, using 2 full arcs and 6-MV photons. Prescribed doses of 70 Gy and 54.25 Gy to the boost (B) PTV and elective (E) PTV (PTV_B/PTV_E) were delivered in 35 fractions. The aim was to deliver 95% of the prescribed doses to 99%/98% of PTV_B/PTV_E, respectively, while limiting the volumes of each PTV receiving >107% of the prescribed dose. A 5-mm transition region (PTV_T) was created between

Outlier removal

In the first iteration of the model cleaning process, 76 OARs were identified with at least 1 of the statistical metrics exceeding the threshold values. Of these, 28 were visually confirmed to be outliers, including 22 and 6 dosimetric and geometric outliers, respectively. In the second iteration, 62 OARs exceeded the thresholds, and 19 were visually confirmed to be outliers, containing 7 and 12 dosimetric and geometric outliers, respectively. Most of the dosimetric outliers were identified in

Discussion

This study revealed that removing geometric and dosimetric outliers from a RP model library did not generally improve resulting plan quality, questioning the necessity for such an extensive process in a suitably large model comprised of consistent plans. Deliberately adding dosimetric outliers to a 70-patient model library resulted in a broadening of the generated DVH prediction range. For 5 to 15 outliers, averaged mean salivary gland doses were marginally affected. The quality of plans

Conclusions

In conclusion, under these study conditions, we have shown that removal of negative dosimetric outliers from a consistent HNC RP model did not improve the quality of plans made by the model. In addition, small gains in OAR sparing were found when 5 or 10 dosimetric outliers were included in a 70-plan model due to placement of the line objective along the inferior boundary of the DVH prediction range by RP. For the same reason, the increase in salivary gland dose of the plans created by the

References (20)

B.E. Nelms et al.
Variation in external beam treatment plan quality: An inter-institutional study of planners and planning systems
Pract Radiat Oncol
(2012)
B. Wu et al.
Fully automated simultaneous integrated boosted-intensity modulated radiation therapy treatment planning is feasible for head-and-neck cancer: A prospective clinical study
Int J Radiat Oncol Biol Phys
(2012)
K.L. Moore et al.
Experience-based quality control of clinical intensity-modulated radiotherapy planning
Int J Radiat Oncol Biol Phys
(2011)
D. Good et al.
A knowledge-based approach to improving and homogenizing intensity modulated radiation therapy planning quality among treatment centers: An example application to prostate cancer planning
Int J Radiat Oncol Biol Phys
(2013)
A. Fogliata et al.
On the pre-clinical validation of a commercial model-based optimisation engine: Application to volumetric modulated arc therapy for patients with lung or prostate cancer
Radiother Oncol
(2014)
J.P. Tol et al.
Evaluation of a knowledge-based planning solution for head and neck cancer
Int J Radiat Oncol Biol Phys
(2015)
M.E. Christianen et al.
Delineation of organs at risk involved in swallowing for radiotherapy treatment planning
Radiother Oncol
(2011)
K.L. Moore et al.
Quantifying unnecessary normal tissue complication risks due to suboptimal planning: A secondary study of RTOG 0126
Int J Radiat Oncol Biol Phys
(2015)
I.J. Das et al.
Intensity-modulated radiation therapy dose prescription, recording, and delivery: Patterns of variability among institutions and treatment planning systems
J Natl Cancer Inst
(2008)
B. Wu et al.
Using overlap volume histogram and IMRT plan data to guide and automate VMAT planning: A head-and-neck case study
Med Phys
(2013)

There are more references available in the full text version of this article.

Cited by (84)

Development and Validation of Single-Optimization Knowledge-Based Volumetric Modulated Arc Therapy Model Plan in Nasopharyngeal Carcinomas
2024, Advances in Radiation Oncology
Knowledge-based planning (KBP) has evolved to standardize and expedite the complex process of radiation therapy planning for nasopharyngeal cancer (NPC). Herein, we aim to develop and validate the suitability of a single-optimization KBP for NPC.
Volumetric modulated arc therapy plans of 103 patients with NPC treated between 2016 and 2020 were reviewed and used to generate a KBP model. A validation set of 15 patients was employed to compare the quality of single optimization KBP and clinical plans using the paired t test and the Wilcoxon signed rank test. The time required for either planning was also analyzed.
Most patients (86.7%) were of locally advanced stage (III/IV). The median dose received by 95% of the high-risk planning target volume was significantly higher for the KBP (97.1% vs 96.4%; P = .017). The median homogeneity (0.09 vs 0.1) and conformity (0.98 vs 0.97) indices for high-risk planning target volume and sparing of the normal tissues like optic structures, spinal cord, and uninvolved dysphagia and aspiration-related structures were better with the KBP (P < .05). In the blinded evaluation, the physician preferred the KBP plan in 13 out of 15 patients. The median time required to generate the KBP and manual plans was 53 and 77 minutes, respectively.
KBP with a single optimization is an efficient and time saving alternative for manual planning in NPC.
Knowledge-based multi-institution plan prediction of whole breast irradiation with tangential fields
2022, Radiotherapy and Oncology
Citation Excerpt :
The scope of this work was to quantify inter-institute variability of KB prediction models for the right whole breast irradiation using tangential fields and to possibly assess the degree of plan transferability between institutes. Quantifying inter-institute planning variability and extending the KB method on a multi-institute scale could have a relevant impact for several applications such as ‘in-silico’ plan comparisons of different techniques as well as plan quality assurance or clinical audit [21–24]. It has also the potential to improve patient selection for whom a more complex technique could be of benefit [19,25].
To quantify inter-institute variability of Knowledge-Based (KB) models for right breast cancer patients treated with tangential fields whole breast irradiation (WBI).
Ten institutions set KB models by using RapidPlan (Varian Inc.), following previously shared methodologies. Models were tested on 20 new patients from the same institutes, exporting DVH predictions of heart, ipsilateral lung, contralateral lung, and contralateral breast. Inter-institute variability was quantified by the inter-institute SD_int of predicted DVHs/Dmean. Association between lung sparing vs PTV coverage strategy was also investigated. The transferability of models was evaluated by the overlap of each model’s geometric Principal Component (PC1) when applied to the test patients of the other 9 institutes.
The overall inter-institute variability of DVH/Dmean ipsilateral lung dose prediction, was less than 2% (20%-80% dose range) and 0.55 Gy respectively (1SD) for a 40 Gy in 15 fraction schedule; it was < 0.2 Gy for other OARs. Institute 6 showed the lowest mean dose prediction value and no overlap between PTV and ipsilateral lung. Once excluded, the predicted ipsilateral lung Dmean was correlated with median PTV D99% (R² = 0.78). PC1 values were always within the range of applicability (90th percentile) for 7 models: for 2 models they were outside in 1/18 cases. For the model of institute 6, it failed in 7/18 cases. The impact of inter-institute variability of dose calculation was tested and found to be almost negligible.
Results show limited inter-institute variability of plan prediction models translating in high inter-institute interchangeability, except for one of ten institutes. These results encourage future investigations in generating benchmarks for plan prediction incorporating inter-institute variability.
Implementation of automatic plan optimization in Italy: Status and perspectives
2021, Physica Medica
To investigate and report on the diffusion and clinical use of automated radiotherapy planning systems in Italy and to assess the perspectives of the community of Italian medical physicists involved in radiotherapy on the use of these tools.
A survey of medical physicists (one per Institute) of 175 radiotherapy centers in Italy was conducted between February 21st and April 1st, 2021. The information collected included the institute’s characteristics, plan activity, availability/use of automatic tools and related issues regarding satisfaction, criticisms, expectations, and perceived professional modifications. Responses were analysed, including the impact of a few variables such as the institute type and experience.
125 of the centers (71%) answered the survey, with regional variability (range: 47%–100%); among these, 49% have a TPS with some automatic option. Clinical use of automatic planning is present in 33% of the centers, with 13% applying it in >50% of their plans.
Among the 125 responding centres the most used systems are Pinnacle (16%), Raystation (9%) and Eclipse (4%). The majority of participants consider the use of automated techniques to be beneficial, while only 1% do not see any advantage; 83% of respondents see the possibility of enriching their professional role as a potential benefit, while 3% see potential threats.
Our survey shows that 49% of the responding centres have an automatic planning solution although clinically used in only 33% of the cases. Most physicists consider the use of automated techniques to be beneficial and show a prevalently positive attitude.
Wide-Scale Clinical Implementation of Knowledge-Based Planning: An Investigation of Workforce Efficiency, Need for Post-automation Refinement, and Data-Driven Model Maintenance
2021, International Journal of Radiation Oncology Biology Physics
Our purpose was to investigate the effect of automated knowledge-based planning (KBP) on real-world clinical workflow efficiency, assess whether manual refinement of KBP plans improves plan quality across multiple disease sites, and develop a data-driven method to periodically improve KBP automated planning routines.
Using clinical knowledge-based automated planning routines for prostate, prostatic fossa, head and neck, and hypofractionated lung disease sites in a commercial KBP solution, workflow efficiency was compared in terms of planning time in a pre-KBP (n = 145 plans) and post-KBP (n = 503) patient cohort. Post-KBP, planning was initialized with KBP (KBP-only) and subsequently manually refined (KBP +human). Differences in planning time were tested for significance using a 2-tailed Mann-Whitney U test (P < .05, null hypothesis: planning time unchanged). Post-refinement plan quality was assessed using site-specific dosimetric parameters of the original KBP-only plan versus KBP +human; 2-tailed paired t test quantified statistical significance (Bonferroni-corrected P < .05, null hypothesis: no dosimetric difference after refinement). If KBP +human significantly improved plans across the cohort, optimization objectives were changed to create an updated KBP routine (KBP’). Patients were replanned with KBP’ and plan quality was compared with KBP +human as described previously.
KBP significantly reduced planning time in all disease sites: prostate (median: 7.6 hrs → 2.1 hrs; P < .001), prostatic fossa (11.1 hrs → 3.7 hrs; P = .001), lung (9.9 hrs → 2.0 hrs; P < .001), and head and neck (12.9 hrs → 3.5 hrs; P <.001). In prostate, prostatic fossa, and lung disease sites, organ-at-risk dose changes in KBP +human versus KBP-only were minimal (<1% prescription dose). In head and neck, KBP +human did achieve clinically relevant dose reductions in some parameters. The head and neck routine was updated (KBP’_HN) to incorporate dose improvements from manual refinement. The only significant dosimetric differences to KBP +human after replanning with KBP’_HN were in favor of the new routine.
KBP increased clinical efficiency by significantly reducing planning time. On average, human refinement offered minimal dose improvements over KBP-only plans. In the single disease site where KBP +human was superior to KBP-only, differences were eliminated by adjusting optimization parameters in a revised KBP routine.
Machine learning applications in radiation oncology
2021, Physics and Imaging in Radiation Oncology
Citation Excerpt :
Early work evaluated the use of a neural network to score radiation therapy treatment plans based on a learning set of physician-scored plans [125]. Knowledge-based planning has also been used for evaluation of treatment plan quality, as compared with that which has been achieved previously [126–129]. This has an application for quality assurance in clinical trials, where plan quality is assessed according to whether it has met the constraints [130,131].
Machine learning technology has a growing impact on radiation oncology with an increasing presence in research and industry. The prevalence of diverse data including 3D imaging and the 3D radiation dose delivery presents potential for future automation and scope for treatment improvements for cancer patients. Harnessing this potential requires standardization of tools and data, and focused collaboration between fields of expertise. The rapid advancement of radiation oncology treatment technologies presents opportunities for machine learning integration with investments targeted towards data quality, data extraction, software, and engagement with clinical expertise. In this review, we provide an overview of machine learning concepts before reviewing advances in applying machine learning to radiation oncology and integrating these techniques into the radiation oncology workflows. Several key areas are outlined in the radiation oncology workflow where machine learning has been applied and where it can have a significant impact in terms of efficiency, consistency in treatment and overall treatment outcomes. This review highlights that machine learning has key early applications in radiation oncology due to the repetitive nature of many tasks that also currently have human review. Standardized data management of routinely collected imaging and radiation dose data are also highlighted as enabling engagement in research utilizing machine learning and the ability integrate these technologies into clinical workflow to benefit patients. Physicists need to be part of the conversation to facilitate this technical integration.
Assessment of a conventional volumetric-modulated arc therapy knowledge-based planning model applied to the new Halcyon© O-ring linac in locoregional breast cancer radiotherapy
2021, Physica Medica
The aim of this study was to evaluate the performance of a knowledge-based planning (KBP) model for breast cancer trained on plans performed on a conventional linac with 6 MV FF (flattening filter) beams and volumetric-modulated arc therapy (VMAT) for plans performed on the new jawless Halcyon© system with 6 MV FFF (flattening filter-free) beams.
Based on the RapidPlan© (RP) KBP optimization engine, a DVH Estimation Model was first trained using 56 VMAT left-sided breast cancer treatment plans performed on a conventional linac, and validated on another 20 similar cases (without manual intervention). To determine the capacity of the model for Halcyon©, an additional cohort of 20 left-sided breast cancer plans was generated with RP and analyzed for both TrueBeam© and Halcyon© machines. Plan qualities between manual vs RP (followed by manual intervention) Halcyon© plans set were compared qualitatively by blinded review by radiation oncologists for 10 new independent plans.
Halcyon© plans generated with the VMAT model trained with conventional linac plans showed comparable target dose distribution compared to TrueBeam© plans. Organ sparing was comparable between the 2 devices with a slight decrease in heart dose for Halcyon© plans. Nine out of ten automatically generated Halcyon© plans were preferentially chosen by the radiation oncologists over the manually generated Halcyon© plans.
A VMAT KBP model driven by plans performed on a conventional linac with 6 MV FF beams provides high quality plans performed with 6 MV FFF beams on the new Halcyon© linac.

View all citing articles on Scopus

: This research was supported by a grant from Varian Medical Systems.

: Conflict of interest: The department has a research collaboration with Varian Medical Systems, and M.D., B.J.S., and W.F.A.R.V. have received honoraria and travel support from Varian medical systems.

View full text

Physics ContributionEffect of Dosimetric Outliers on the Performance of a Commercial Knowledge-Based Planning Solution

Purpose

Methods and Materials

Results

Conclusions

Introduction

Section snippets

Methods and Materials

Outlier removal

Discussion

Conclusions

Pract Radiat Oncol

Int J Radiat Oncol Biol Phys

Int J Radiat Oncol Biol Phys

Int J Radiat Oncol Biol Phys

Radiother Oncol

Int J Radiat Oncol Biol Phys

Radiother Oncol

Int J Radiat Oncol Biol Phys

Intensity-modulated radiation therapy dose prescription, recording, and delivery: Patterns of variability among institutions and treatment planning systems

J Natl Cancer Inst

Using overlap volume histogram and IMRT plan data to guide and automate VMAT planning: A head-and-neck case study

Med Phys

Physics Contribution
Effect of Dosimetric Outliers on the Performance of a Commercial Knowledge-Based Planning Solution