Offline and online LSTM networks for respiratory motion prediction in MR-guided radiotherapy

To obtain motion trajectories from the cine MRIs we used an in-house developed software. Figure 1 summarizes the workflow of the motion extraction from the cine MR frames containing target (green) and boundary (red) contours (a). Briefly, both target and boundary contours were extracted from the videos using thresholds in RGB-space (b). The contours were then filled using the watershed algorithm (Roerdink and Meijster 2000). From the filled contours we subsequently computed the superior–inferior (SI) tumor centroid position relative to the fixed boundary SI centroid position (c). Once the tumor SI motion curves (d) were obtained for all patients, further pre-processing was done, as detailed in the following sections.

Zoom In Zoom Out Reset image size

First, we replaced outliers arising from incorrect filling of contours using sliding windows of size three. In detail, we computed the median centroid position within the current sliding window and if the absolute difference between the central data point in the window and the median value was a larger than an optimized threshold, we replaced that point with the median value of the window. For this step, the curves were temporarily normalized such that a single threshold independent from the absolute motion amplitudes could be used. The normalization was then reversed as the motion in mm is needed to exclude cine videos with small motion (see 2.2.4). After that, the curves were smoothed with a moving average filter applied on sliding windows of size three.

For the LMU cohort only, we then analyzed the motion trajectories to detect the breath-holds. Using sliding windows of size 20, we considered the current window a breath-hold if the median deviation from the median centroid position of the window was smaller than an optimized threshold. This information was then used to exclude all breath-hold data points from the motion trajectories and keep only the free-breathing sub-trajectories in between. Additionally, for this cohort we separated the data according to detected imaging pauses. Cine imaging pauses are inherently part of the MRIdian MR-linac treatment: when the gantry rotates from one irradiation angle to the next, its moving electronics interferes with the MRI causing the image quality to degrade. These degraded cine MRI frames are automatically excluded from the exported videos by the vendor, but their start is indicated by displaying the statement 'imaging paused' on the top right of the video. We automatically detected the frames where this statement was displayed and used this information to separate the motion trajectories into two sub-trajectories. This avoids jumps in the curves arising from an imaging pause between data points.

For both the LMU and the Gemelli cohorts we excluded all data of cine videos for which the interquartile range (IQR) of SI free-breathing motion was below 3.5 mm (in-plane resolution of MRIdian cine MRIs), as this motion is more substantially affected by imaging noise. This led to the exclusion of 73 LMU and 5 Gemelli videos. Finally, we again normalized all motion curves to the range −1 to +1 using the minimum and maximum tumor centroid position of each cine MRI. These min/max values were saved to disk and used during evaluation to undo the normalization of the predicted curves.

After pre-processing, we obtained 16.1 h of motion data without breath-holds (105.8 h if the breath-holds were not excluded) for the LMU cohort and 1.5 h of free-breathing motion for the Gemelli cohort.

Following the terminology used by Remy et al (2021), motion prediction is about obtaining future target positions (at time t + Δt) from the current target motion (at time t). In general, for a given time step i, every prediction task can be simply formulated as

$$\begin{eqnarray}&&{\hat{{\boldsymbol{y}}}}_{i}=f({{\boldsymbol{x}}}_{i}),\end{eqnarray} \tag{ 1 }$$

where f() is a motion prediction algorithm, x _i is the ith vector containing the input data window and ${\hat{{\boldsymbol{y}}}}_{i}$ is the ith vector with the predicted output data window. The corresponding vector y _i contains the ground truth output data window used to optimize the algorithm. In our case, x and y contain input and output SI target centroid positions. The length of x was treated as a hyper-parameter (see section 2.4.2). On the other hand, the length of $\hat{{\boldsymbol{y}}}$ (and y ) is automatically related to the forecasted time span. In this study, we investigated forecasts of 250 ms, 500 ms and 750 ms, corresponding to $\hat{{\boldsymbol{y}}}$ having length of 1, 2 or 3, respectively, for 4 Hz imaging.

Over the last decade, several motion prediction algorithms have been proposed to account for latencies in image guided RT. Joehl et al used motion traces from a robotic radiosurgery system to compare 18 different predictors for 160 ms and 480 ms forecasts (Joehl et al 2020). On average, they found an (online) LR model to perform best, so we decided to leverage it as baseline model. Mathematically, the regression function is defined as (Krauss et al 2011)

$$\begin{eqnarray}&&f({\boldsymbol{x}})={{\boldsymbol{\beta }}}^{T}{\boldsymbol{x}}+{\beta }_{0},\end{eqnarray} \tag{ 2 }$$

where the vector β contains the parameters of the regression model. The loss function to be minimized to solve the regression is given by

$$\begin{eqnarray}&&L({\boldsymbol{\beta }})=\sum _{i=1}^{N}{\left({y}_{i}-f({{\boldsymbol{x}}}_{i})\right)}^{2}+\lambda | | {\boldsymbol{\beta }}| {| }^{2},\end{eqnarray} \tag{ 3 }$$

where N is the number of input/output training windows and λ is an L2-regularization parameter. If λ ≠ 0, the term ridge regression is usually used.

If we define a matrix X such that its rows equal the input window vectors x _i and Y a matrix with the true output window vectors y _i , the loss function L( β ) is analytically solved by the optimal parameters

$$\begin{eqnarray}&&{{\boldsymbol{\beta }}}^{* }={\left({{\boldsymbol{X}}}^{T}{\boldsymbol{X}}+\lambda {\boldsymbol{I}}\right)}^{-1}{{\boldsymbol{X}}}^{T}{\boldsymbol{Y}},\end{eqnarray} \tag{ 4 }$$

where I is the identity matrix. Therefore, the LR model has a closed form solution and does not need iterative optimization.

Recurrent neural networks (RNN) are a class of machine learning algorithms ideally suited for sequential input data (e.g. time series data). Compared to artificial neural networks or convolutional neural networks, recurrent neural networks have an extra set of weights which connects hidden layers from one time point to the next. However, the original RNN module comprising a simple fully connected layer was found to be limited due to unstable gradient issues when back-propagating over longer input sequences (Shen et al 2020). Therefore, more advanced RNN architectures have been proposed and the most widely adopted is the LSTM model (Hochreiter and Schmidhuber 1997), which was specifically designed to more easily learn longer sequences of data.

The repeating module of LSTMs is shown in figure 2. LSTMs introduce the memory cell state c ^t , which allows a stable back-propagation of errors and straightforward flow of information. To remove or add information to the cell state, structures called gates are used. Intuitively, the forget gate f ^t is used to keep/discard past information in c ^t−1 while the input gate i ^t allows to filter information in the new memory cell state ${\tilde{{\boldsymbol{c}}}}^{t}$. The previous gates and memory are then combined to build the final memory cell state c ^t . The final memory cell is filtered with the output gate o ^t to keep/discard some information and build the hidden state h ^t . Mathematically, at a specific time step t, the LSTM module is described as follows:

$$\begin{eqnarray}&&{\rm{Forget}}\,{\rm{gate}}:\qquad {{\boldsymbol{f}}}^{t}=\sigma ({{\boldsymbol{W}}}_{f}{{\boldsymbol{x}}}^{t}+{{\boldsymbol{U}}}_{f}{{\boldsymbol{h}}}^{t-1}+{{\boldsymbol{b}}}_{f})\end{eqnarray} \tag{ 5 }$$

$$\begin{eqnarray}&&{\rm{Input}}\,{\rm{gate}}:\qquad {{\boldsymbol{i}}}^{t}=\sigma ({{\boldsymbol{W}}}_{i}{{\boldsymbol{x}}}^{t}+{{\boldsymbol{U}}}_{i}{{\boldsymbol{h}}}^{t-1}+{{\boldsymbol{b}}}_{i})\end{eqnarray} \tag{ 6 }$$

$$\begin{eqnarray}&&{\rm{New}}\,{\rm{memory}}\,{\rm{cell}}\,{\rm{state}}:\qquad {\tilde{{\boldsymbol{c}}}}^{t}=\tanh ({{\boldsymbol{W}}}_{c}{{\boldsymbol{x}}}^{t}+{{\boldsymbol{U}}}_{c}{{\boldsymbol{h}}}^{t-1}+{{\boldsymbol{b}}}_{c})\end{eqnarray} \tag{ 7 }$$

$$\begin{eqnarray}&&{\rm{Final}}\,{\rm{memory}}\,{\rm{cell}}\,{\rm{state}}:\qquad {{\boldsymbol{c}}}^{t}={{\boldsymbol{f}}}^{t}\odot {{\boldsymbol{c}}}^{t-1}+{{\boldsymbol{i}}}^{t}\odot {\tilde{{\boldsymbol{c}}}}^{t}\end{eqnarray} \tag{ 8 }$$

$$\begin{eqnarray}&&{\rm{Output}}\,{\rm{gate}}:\qquad {{\boldsymbol{o}}}^{t}=\sigma ({{\boldsymbol{W}}}_{o}{{\boldsymbol{x}}}^{t}+{{\boldsymbol{U}}}_{o}{{\boldsymbol{h}}}^{t-1}+{{\boldsymbol{b}}}_{o})\end{eqnarray} \tag{ 9 }$$

$$\begin{eqnarray}&&{\rm{Hidden}}\,{\rm{state}}:\qquad {{\boldsymbol{h}}}^{t}={{\boldsymbol{o}}}^{t}\odot \tanh ({{\boldsymbol{c}}}^{t}),\end{eqnarray} \tag{ 10 }$$

where b , W and U denote the biases, input window weights and recurrent weights which are learned during the optimization process. The symbol ⊙ represents element-wise multiplication between matrices/vectors. The sigmoid function σ(x) was used for the gates and the hyperbolic tangent function was used to generate the states. For each time step, the hidden state of one LSTM layer is used as input for the next LSTM layer. For the last hidden layer of the LSTM, the hidden state of the last time point t_f is input to a fully connected layer to build the predicted output window

$$\begin{eqnarray}&&{\rm{Predicted}}\,{\rm{output}}:\qquad {\hat{{\boldsymbol{y}}}}_{i}={{\boldsymbol{W}}}_{\mathrm{FC}}{{\boldsymbol{h}}}^{{t}_{f}}+{{\boldsymbol{b}}}_{\mathrm{FC}},\end{eqnarray} \tag{ 11 }$$

where W _FC and b _FC denote the weight matrix and bias vector for the fully connected layer.

Zoom In Zoom Out Reset image size

In this study, a stateless LSTM was implemented, which means that during optimization the hidden state and the cell state were cleared after every batch of data. The LSTM architecture was inspired by the one used by Lin et al (2019). Specifically, we performed our hyper-parameter optimization based on the range of values used in their hyper-parameter search. More details can be found in section 2.4. Figure 3 schematically shows the working principle of the proposed LSTM. At every time point, a single SI tumor centroid position is given as input for as many points as the length of the input data window. The LSTM modules in the hidden layer (green boxes) process the time-dependent information as shown in figure 2 until the last LSTM module is reached. The hidden vector which is output by the last LSTM module is mapped via a fully connected layer to the predicted output window following equation (11). Note that in figure 3 only one hidden layer is shown whereas this number was treated as a hyper-parameter in our optimizations (see section 2.4.2).

Zoom In Zoom Out Reset image size

To optimize and evaluate the models we split the LMU data into training, validation and testing sets. Specifically, we assigned the motion trajectories belonging to 60% of the patients to the training set (52 patients), 20% to the validation set (18 patients) and the remaining 20% to the testing set (18 patients) and did this procedure only once at the beginning. This splitting roughly also led to 60% of the motion trajectories being in training (9.1 h), 20% in validation (4.0 h) and 20% in testing (3.0 h). As the Gemelli cohort was smaller (1.5 h) but at the same time in free-breathing, we decided to use this dataset as an independent additional testing set. Finally, we also applied the best models trained/validated on the LMU data without breath-holds to the LMU testing set without excluding the breath-holds during pre-processing.

To find the optimal set of hyper-parameters for both the LSTM and the LR we repeatedly performed training and validation while varying the parameters for all three analyzed forecasts and for all four training strategies (see section 2.4.3) separately. For the LSTM, the following hyper-parameters were varied, based on the hyper-parameter search performed by Lin et al:

• Number of layers: the number of hidden layers of the LSTM was chosen among the following values {1, 3, 5, 10}.
• Dropout: the dropout rate on the outputs of each hidden layer (but the last one) was sampled from the set {0, 0.1, 0.2}.
• Learning rate: the learning rate of the optimizer was sampled from the set {1 × 10⁻⁴, 5 × 10⁻⁴, 1 × 10⁻³, 5 × 10⁻³, 1 × 10⁻²}.
• Batch size: the batch size, i.e. the number of input windows fed to the network simultaneously, can be varied only for the offline trained LSTM (see section 2.4.3) and was set to either 64 or 128.
• L2-regularization: the L2-regularization parameter λ (also called weight decay) was sampled from the set {0, 1 × 10⁻⁶, 1 × 10⁻⁵, 1 × 10⁻⁴}.

All optimizations for the LSTMs were carried out using the Adam optimizer (Kingma and Ba 2015) with a normalized mean squared error (MSE) loss function and learning rates from the set shown above. We set the number of features in the hidden layer vector h ^t to 15 like Lin et al. No batch normalization was used. For the LR, the following hyper-parameter was varied in logarithmic steps over a large range of values:

• L2-regularization: the L2-regularization parameter λ was sampled from the set {1 × 10⁻⁵, 1 × 10⁻⁴, 1 × 10⁻³, 1 × 10⁻², 1 × 10⁻¹, 1, 10}.

Both for the LSTM and for the LR we varied the length of the input data window x between 8, 16, 24 and 32 data points, corresponding to 2, 4, 6 and 8 seconds of past motion.

Retraining models on recent motion data has been shown to improve the predictive performance (Krauss et al 2011, Sun et al 2020). Thus, two different training strategies were investigated for the LSTM and for the LR model.

• Offline LSTM: the offline LSTM optimization was carried out following the typical machine learning training/validation/testing subdivision. The model was iteratively optimized on the training set for 600 epochs while monitoring training and validation losses. If the validation loss improved, the model weights were saved to disk. If the validation loss did not improve for 100 epochs, the optimization was stopped, a technique know as early stopping. For final inference, we loaded the weights and hyper-parameters of the best performing model on the validation set and applied it unchanged to the testing set.
• Offline+online LSTM: to allow adaptation to recent motion patterns, we continuously retrained the LSTM on current data. Specifically, we first loaded the weights of a previously optimized offline LSTM. The LSTM was then re-optimized on the last 20 s of validation data using a sliding set of validation input/output windows updated with a first-in-first-out approach, as shown in figure 4. The online optimization of the LSTM was done for 10 epochs, taking about 150 ms. This would allow an implementation in a 4 Hz image acquisition clinical scenario. To prevent the iterative optimization to introduce an additional latency, within the 250 ms between one cine MRI frame and the next, we performed the prediction before optimizing the LSTM. To calculate the validation loss we used the ground truth data point lying 250 ms, 500 ms or 750 ms (depending on the forecast) in the future with respect to the last centroid position in the currently used 20 s of optimization data. For final inference, we loaded the offline LSTM, set the hyper-parameters leading to the best result on the validation set and continuously retrained and evaluated the model on the testing set.
• Offline LR: the offline LR training is analogous to the offline LSTM training but for the fact the the LR is solved analytically while the LSTM is iteratively optimized. Specifically, the LR was solved on the training set and then applied unchanged to validation set to perform the hyper-parameter search. For final inference, as for the offline LSTM, we loaded the weights and set the hyper-parameters of the best performing model on the validation set and applied it unchanged to the testing set.
• Online LR: on the other hand, the online LR is different from offline+online LSTM. As no iterative fine-tuning is needed for the LR, no weights from a pre-trained offline LR were loaded. The online LR was continuously solved 'from scratch' based on the last 20 s of validation data using a sliding set of validation input/output windows updated with a first-in-first-out approach (figure 4). As solving the LR is simply a matrix multiplication (see equation (2)), it takes less than 1 ms. As this additional latency is not significant, for the online LR we performed the prediction after the optimization, as illustrated in figure 4. This is advantageous as the model's prediction can take into account the most recently acquired data point.

Zoom In Zoom Out Reset image size

As mentioned in section 2.3.1, our data was subdivided in input windows x where the number of entries len( x ) is a hyper-parameter. To obtain a set of windows with a total duration of 20 s to be used for online training, we need several input windows. Given that every input window is shifted by one and that the cine imaging is performed at a frame rate of 4 Hz, the number of input windows needed is given by

$$\begin{eqnarray}&&{\rm{Nr.}}\,{\rm{input}}\,{\rm{windows}}=20\cdot 4-\mathrm{len}({\boldsymbol{x}})-1.\end{eqnarray} \tag{ 12 }$$

Therefore, 73 input windows are needed if we choose len( x ) = 8 as in figure 4. This number corresponds to the batch size for the offline+online LSTM, which is thus not freely selectable if we fix the duration of the online optimization data to 20 s. The number of input windows between the last window used for optimization and the window used for prediction is given by the current forecasted time span. For example for the 500 ms forecast scenario shown in figure 4, this difference is equal to two (e.g. step 0: x ₇₂ is the last window in the optimization matrix and x ₇₄ is the window used for prediction). For the 250 ms forecast this difference is one and for the 750 ms forecast this difference is equal to three.