Radiomics in radiation oncology—basics, methods, and limitations

To calculate radiomics features, manual or semi-automatic segmentations of the region of interest (ROI) or volume of interest (VOI) are mandatory. Typically, the contrast-enhancing portion of the tumor in MRI is used for radiomics analysis. However, including information contained in the infiltration zone of the lesion by also considering signal abnormalities on T2-weighted or fluid-attenuated inversion recovery (FLAIR) MRI provides a more realistic representation of the whole tumor and allows the radiomics analysis to be performed on a larger segment, potentially encoding more information and resulting in a better diagnostic performance. Although the number of studies using amino acid PET images for radiomics analysis is still low, especially the combined analysis of amino acid PET and MRI radiomics encodes more diagnostic information than either modality alone [13, 14] and might gain clinical relevance. In patients with brain tumors, image segmentation in clinical routine is usually performed manually on CT or MRI for the purpose of radiotherapy planning or volumetric assessment of therapy response. However, a manual, three-dimensional differential segmentation of tumor regions with contrast enhancement, necrosis, and perifocal edema is laborious, time consuming, and strongly dependent on the performing physician. Methods from the field of AI including textural feature analysis and deep learning-based methods are already available and currently under investigation for routine clinical application [15‐21].

As mentioned above, radiomics aims at the extraction of quantitative imaging features from routinely acquired imaging data. Consequently, in order to enable a high reproducibility and generalizability of the results, the image data have to undergo several preprocessing steps before feature extraction. One of the first preprocessing steps is interpolation of the imaging data to isotropic voxel spacing, which allows for a better comparison of heterogenous, multi-institutional imaging data. Furthermore, the calculation of radiomics features, especially textural features, requires rotationally invariant voxels, achieved by interpolation. Images can either be upsampled, e.g., the original image with a voxel spacing of 1.0 × 1.0 × 3.0 mm³ is upsampled to 1.0 × 1.0 × 1.0 mm³, or downsampled, e.g., the original image with a voxel spacing of 1.0 × 1.0 × 3.0 mm³ is downsampled to 3.0 × 3.0 × 3.0 mm³. While upsampling introduces artificial information and might increase image noise, downsampling conversely incurs information loss. Consequently, there is currently no clear recommendation for either of the two procedures [22].

Discretization or quantization of image intensities is of particular importance to allow for comprehensible feature extraction [22]. Two methods for image discretization are commonly used. The first method performs a discretization of the image intensities to a fixed number of bins, which allows for a direct comparison of feature values across different patients and partly performs an image normalization, which is of importance for imaging procedures such as structural MRI that are usually acquired in arbitrary units. However, no correlation to the original image intensities can be established. The second method uses a fixed bin size, whereby a new bin is assigned for every intensity interval with a fixed bin with. Importantly, for structural MRI data with arbitrary intensity units, the fixed bin size discretization is not recommended. However, as the relationship to the original intensity scale is maintained, the fixed bin size discretization could be especially useful for quantitative imaging modalities such as PET. Of note, the image discretization has a substantial impact on the extracted radiomics features and, hence, on the reproducibility of the results [22].

Normalization of the image intensities ensures a better comparability of the results between different scanners, protocols, and patients. Commonly used procedures for image intensity normalization are white-stripe [23] or z‑score normalization [24]. Other typical preprocessing steps include, but are not limited to, spatial smoothing, noise reduction, spatial resampling, brain extraction, and corrections of MRI field inhomogeneities.

Following image preprocessing and segmentation, radiomics features can be calculated, most of which reflect tumor heterogeneity. Since the radiomics features are mathematically predefined and based on a huge number of slightly different mathematical definitions, a large number of radiomics features (usually more than 1,000) can be extracted from medical images. Typically, radiomics features are divided into the following subgroups:

In summary, a large number of features can be calculated from a single segmented lesion, leading to the problem of distinguishing the parameters relevant to the clinical problem under investigation from the irrelevant and redundant ones. This so-called feature selection is of high importance for generating a meaningful predictive or prognostic model from the computed features, especially if the number of available datasets is limited. Feature selection uses advanced statistical methods to identify a subset of features that are neither redundant, constant, duplicated, irrelevant, nor highly correlated [12].

Generally, two types of feature selection techniques are used in radiomics studies, unsupervised and supervised feature selection [29]. Unsupervised feature selection methods such as principal component analysis (PCA) or cluster analysis aim at the identification and removal of redundant features from the feature space, whereby class labels are not considered [12]. Supervised feature selection techniques also take into consideration the relation of the features to the class labels, i.e., features that contribute most to the diagnostic problem are preferred. Consequently, supervised feature selection techniques usually result in better feature subsets compared to the unsupervised methods. Different unsupervised feature selection methods exist, of which filter methods, wrapper methods, and embedded methods are those most prominent in radiomics.

Filter methods are also called univariate methods and statistically evaluate the relation between the features without considering their correlations and interactions. Univariate methods are, for example, the chi-squared score, the Student’s t‑test, the Wilcoxon rank sum test, or the Fisher score [30, 31].

Wrapper methods, also called multivariate methods, partly overcome the limitations of univariate methods by taking into consideration correlations and interactions among the radiomics features. While univariate methods only investigate the statistical relationship between the radiomics features, wrapper methods create a subset of features, apply this subset to a predictive model, and evaluate the quality of its performance. Thereafter, a new subset of features is tested and, finally, the best performing subset of features represents the final set of selected features. Due to the iterative nature of wrapper methods, these methods are computationally intensive. These methods include bidirectional search, exhaustive feature selection, forward feature selection, or backward feature elimination [30, 31].

Embedded methods combine the advantages of filter and wrapper methods. The feature selection process is performed during the generation of the machine learning model, i.e., during the training phase of the model. Here, interactions and correlations of the radiomics features are considered, leading to more accurate feature selection results compared to filter methods. Additionally, since feature selection is performed during the training phase of the machine learning models and does not require an additional predictive model solely for performance evaluation of the different subsets of features, embedded methods are computationally faster than wrapper methods and less prone to overfitting. Examples of embedded feature selection methods are tree-based algorithms such as the random forest classifier, the least absolute shrinkage and selection operator (LASSO), or ridge regression [30, 31].

Now that a subset of relevant features with low redundancy has been identified by feature selection, a mathematical model for the prediction of a known, underlying ground truth can be generated. This step within the radiomics workflow is called model generation. Usually, several different machine learning algorithms and classifiers can be used to generate predictive or prognostic models according to the goal of the study. Among the machine learning algorithms and classifiers most commonly used for radiomics analysis are linear and logistic regression, decision trees, e.g., random forests, neural networks, support vector machines, or the Cox proportional hazards model in case of censored survival data. Model generation includes the iterative search for a set of optimal parameters that define the general structure of the model, a process called hyperparameter tuning. In order to identify the best possible machine learning algorithm for the diagnostic problem, the selected model has to be evaluated on a subset of data. To avoid the risk of overfitting by generating and testing the mathematical model on the same subset of data, the available dataset is ideally subdivided into a training and a validation dataset. If the model was trained and evaluated on the same subset of data, a perfect classification of results could be easily achieved by an algorithm that simply repeats the labels of the training data. Obviously, such a highly overfitted model would not provide any useful prediction on new data that was not part of the training dataset. It should be noted, that the distribution of samples of each class remains approximately the same after data splitting, i.e., if 40% of samples were diagnosed with a recurrent tumor, and 60% were diagnosed with treatment-related tissue changes in the original dataset, this proportion of diagnoses should also remain approximately the same in the training and the validation dataset (stratified split of data), which is particularly important for small datasets. Ideally, the model that showed the best diagnostic performance in the validation dataset is finally applied to a test dataset. The test dataset represents the data the model would face when applied in clinical routine. Consequently, every radiomics model should prove its performance, robustness, and reliability on the test dataset, but the test dataset should never be used for tuning of  model hyperparameters.

The described workflow for model generation and evaluation including splitting of the data into three subgroups obviously requires large datasets. Unfortunately, oncological studies including studies in the field of radiation oncology usually contain a maximum of several hundred datasets. However, machine learning also offers methods such as bootstrapping or cross-validation to assess model performance even without the availability of a test dataset. In cross-validation, e.g., 10-fold cross-validation, the dataset is partitioned into ten subsets of equal size. The generated model is evaluated on nine datasets as training data while one subset of data is retained for model validation. This process is repeated ten times, with each subset used once as validation data. Finally, to assess the overall model performance, the classification accuracy from each iteration is averaged.

Deep learning generally describes the process of applying deep neural network architectures for problem solving, which are particular types of machine learning algorithms. Originally, artificial neural networks that provide the basis for more advanced architectures were inspired by the working principle of the human visual system. By adding layers of hidden neurons beyond the simple input and output layers from artificial neural networks, a new level of complexity was added that enabled deep learning. Deep learning-based radiomics automatically identifies and extracts high-dimensional features from the input images at different levels of scaling and abstraction, resulting in models especially useful for pattern recognition or classification of high-dimensional non-linear data [32]. The usefulness of deep learning-based methods for the automated identification and segmentation of brain metastases and gliomas for radiation therapy planning has been demonstrated in several studies [17‐21].

Deep learning-based radiomics uses a workflow that is very different from the feature-based radiomics approach described above. Instead of using mathematically predefined features, different architectures of neural networks such as convolutional neural networks (CNNs) or auto-encoders are used to generate and identify the most important features from the input data. In particular, autoencoder networks, which are a special, unsupervised variant of CNNs, aim at compression of the image content and mapping onto a relatively short but representative feature vector [33]. In general, deep learning-based radiomics uses a cascaded system of single-layer neural networks which are trained to learn and identify structures of relevance for the classification problem in the imaging data [34]. Here, previous mathematical definitions of features and feature selection become unnecessary. Further combinations of the extracted feature vectors are then combined to generate features with an even higher level of abstraction. In a final step, the identified features can be used for classification by the neural network itself, or leave the network and undergo the process of model generation similar to the feature-based radiomics approaches described above using different classifiers from conventional machine learning such as decision trees, regression models, or support vector machines. Of note, while feature-based radiomics always requires segmented images for feature extraction, deep learning-based radiomics, especially CNNs, also function on unsegmented images.

Since the features are generated and extracted directly from the underlying data and a subset of best performing features is automatically extracted, feature selection is rarely performed. However, in order to reduce the risk of overfitting, regularization techniques and dropout of learned connection weights are used. One limitation of deep learning-based radiomics is the high correlation between the features and the input data, as the features are generated from that very data. Therefore, in contrast to feature-based radiomics, large datasets are necessary to identify a relevant and robust feature subset. However, using a machine learning technique called transfer learning, this limitation can be partly overcome. In transfer learning, a neural network is used that has already been pre-trained on a different, but closely related task, e.g., a neural network for brain tumor segmentation that was originally trained on imaging data from patients with brain metastases might also provide useful results for the segmentation of glioma patients [35]. Hereby, both the amount of data necessary to identify a relevant feature subset and the computational demand are reduced.