Improving preoperative prediction of surgery duration

The data for this study was extracted from one of the major metropolitan hospitals in Australia. The hospital comprises 783 beds, 15 operating theatres that are available 24 h/day, 5 endoscopy suites, 2 interventional theatres, a dedicated emergency obstetric theatre, 20 preoperative holding bays and 37 post anaesthetic care unit bays. The data was extracted from the Theatre Management System (Department of Health, Government of Western Australia) which is populated exclusively by interfaces with the web-based Patient Administration System (webPAS) and included over 70,000 de-identified records of patients undertaking elective surgeries (i.e., planned surgeries), from November 2014 to June 2020. The study was approved by the FSH QI Medical Anaesthesia & Pain Medicine Committee (Quality activity 29,238) and CSIRO Health and Medical Human Research Ethics Committee (2019_024_LR).

The main aim of this study was to predict the operation duration which is defined as the total minutes from patients entering the OR to patients leaving the OR. To that end, several steps were taken to prepare the data for modelling as follows: (1) operations with multiple records were excluded, (2) surgeries with operation duration recorded as 0 or less were removed, (3) operations with missing values for key variables including specialty and procedures were trimmed, (4) operations that included procedures with less than 10 records within the entire dataset were filtered out. In this study, we assumed that the data is missing at random (MAR). The ASA score (reflecting patient frailty) is one of the candidate variables, however, values were missing for almost 40% of the records. Since the ASA score is a categorical feature, we used the mode formula (i.e., selecting the most frequent value in the dataset) to fill in the missing values [19]. More specifically, for each operation, if it included a missing ASA score, the missing value was replaced by the most frequent ASA score among the previous operations to that date with the same set of procedures. With this method, we reduced the number of records missing the ASA score to only 7.7%. In this study, we conducted a sensitivity analysis to explore alternative scenarios for missing data. We created multiple datasets after applying multiple imputation procedures such as mode, median, last observation comes forward (LOCF) and KNN [23]. In addition, we also created another dataset in which all missing values were filtered out, i.e., only keeping the complete records, and it was confirmed that the results remained similar under alternative strategies. After cleaning the data, 52,171 elective operation records were available for developing predictive models. Figure 1 shows the process of cleaning data and the removed records as a result of each condition.

To evaluate the predictive models we split the data into training and testing data based on the year of operations with the intention of simulating real-world settings where past records are used to predict upcoming events in the future [7, 14, 24‐27]. The training dataset included elective operations from November 2014 to June 2019, a total of 41,794 operations, and the testing dataset contained operations from July 2019 to June 2020, a total of 10,377 operations. Also, because of its right skewness, as can be seen in the top plot in Fig. 2, and to improve the accuracy of the prediction models, we applied a BoxCox transformation of the surgery duration times to more closely resemble a normal distribution. As noted in [28], this transformation can be useful for many practical problems including clinical data. It is also worth mentioning that although using log-normal distribution is the most common method to transform operation duration [29, 30], as suggested by [31] we used the BoxCox transformation since log-normal distribution is actually a special case of the BoxCox transformation. Also, the data skewness after log-normal transformation is 0.12, however, the skewness after BoxCox is -0.00062 which means the BoxCox method transforms the operation duration into a more favourable symmetric distribution. Note that, after predicting the duration of surgery, the predicted values were transformed back for accuracy calculations and comparison with the actual values of the datasets. The histogram of the operation times using the given actual data and the transformed version are shown in Fig. 2.

Figure 3 shows the distribution of the operation durations (i.e., response variable), across different medical specialties, with at least 15 records. This figure also captured the number of data records contained in each specialty and the average operation time for each specialty, shown by the red colour line.

The predictor variables, used for model development, were selected after an in-depth review of available data sources. First, we focused on variables that are available preoperatively, several days before the operation. Several variables are commonly used in the existing models in the literature and also were available in our dataset including “Order” (e.g., operation order in session), “Session” (e.g., morning or afternoon), “Theatre” (i.e., operating room number), and “Cancellation” (i.e., the cancellation status of the previous operation). However, they were not part of our model development as they would not be relevant in a potential implementation, even though they may improve the model’s accuracy.

Besides the above-mentioned features, we investigated several other common features including “age”, “gender”, “patient history” (i.e., the previous number of surgeries with the same procedures), and “admission type”. But they were also excluded from model development as they either did not improve the model or slightly made the model worse. These findings are consistent with reports from others [14, 32]. It is also worth mentioning that the data used in this study was collected from an OR management database that lacked the information related to the surgeon’s estimation of the operation duration as well as the surgical team (e.g., the surgeon, number of anaesthetists and nurses) was not available. The features used for model development in this study are shown in Table 1. All five predictors were nominal variables.

The first three features listed in Table 1 (i.e., Specialty, Number of procedures, Procedure combination) are operation-related factors, while the other two are patient characteristics. The CCI is an indicator of the severity and complications of a patient’s condition which uses 17 different patient diagnoses (e.g., diabetes, cancer, liver disease) to calculate the index. Since the calculation of the CCI score needs a thoroughly accurate review of medical records collected over years before hospital admission [33], we considered this feature as it reflects the patient’s medical history. In this paper, the comorbidity package [34] in R was used for computing CCI using the International Classification of Diseases, 10th Revision (ICD-10). It is also worth mentioning that the Present On Admission (POA) diagnoses were used when calculating CCI. The CCI was categorised into three groups: 0, 1–2, and 2+ [35]. The ASA score is a six-level scale, from 1 to 6, that measures the fitness and physical condition of the patient before surgery and anaesthesia. The distribution of the patient characteristics in the given dataset is shown in Table 2.

Several machine learning (ML) models have been applied to investigate their efficiency and accuracy in predicting surgical operation time: Linear Regression (LR), Polynomial Regression (PR), Random Forest (RF), Extreme Gradient Boosting (XGBoost), and Nearest Neighbours Regression (NNr).

Linear Regression (LR) is a prediction approach that tries to find the best relationship between features (i.e., independent variables) and the response variable by fitting a line to the given data sets [36]. The LR model assigns one scale factor to each feature, called a coefficient, and also one extra coefficient, called the intercept, to give the model an additional degree of freedom. The number of coefficients determines the complexity of the model. Although LR models are easier to implement and interpret, they fail on complex datasets and are quite sensitive to outliers. Polynomial regression (PR) [37] is a form of linear regression. The main difference between PR and LR is that, in PR, the relationship between features and response variable is non-linear (i.e., curvilinear).

Random Forest (RF) [38] is a machine learning algorithm that can be applied for both classification (i.e., the response variable is nominal) and regression (i.e., the response variable is continuous) problems. The RF approach is constructed from decision trees [39] by combining multiple small trees on various subsets of the given dataset to create a ‘forest’. Each tree produces its outputs, and the RF algorithm calculates the average of the outputs of all trees. The main advantage of RF algorithms is that they handle large datasets and also datasets with a great portion of missing values.

Extreme Gradient Boosting (XGBoost) [40] is another ensemble technique based on a decision tree algorithm similar to the RF algorithm. However, unlike the RF where all trees are built at the same time in parallel and each is trained independently, XGBoost builds one tree at a time sequentially and uses information from previous trees to improve subsequent trees. The idea is to correct the previous mistake made by the model, learn from it and in the next step, improve the performance. Therefore, XGBoost is greedier than the RF algorithm.

Nearest Neighbours Regression (NNr) [41] divides the data into different ‘K’ neighbourhoods (i.e., classes) using feature similarities (i.e., distances). Then, NNr assigns the new data points to the neighbourhood that is the most similar to the available ones. The NNr doesn’t perform any training, calculations, or building models until a new prediction is performed. The main advantage of NNr is that it is fast and allows users to understand and interpret the model. However, NNr struggles to adapt to highly complex relationships between features and response variables.

As already mentioned in "Introduction" section , the most common approach to estimating the duration of surgery is simply taking the average of previous records for the same procedure type [13]. The hospital studied in this paper also uses a similar approach by including surgeon information as well as the procedure type. At the study site, emergency surgery bookings are taken from the surgeon’s estimation of what they have entered on the booking request and estimates of emergency surgery duration can change upon findings duration the operation. Elective surgery relies on the use of a surgical estimator tool to prevent overruns and underutilisation, hence its vital to have historical information inform procedural scheduling in addition to the proceduralists’ estimates.

In this study, we had no access to the estimated procedure durations based on the above-mentioned approaches. In addition, surgeon-related information was not also available in our data, which made it almost impossible to reproduce surgery duration estimations based on the current approach at the study hospital. However, we provided two simple estimations using similar approaches to those used in the hospital: “Mean5” and “MeanAll”. The Mean5 approach takes the average of the previous 5 records of operation durations with the same procedures involved in that operation. The MeanAll approach is similar to Mean5 but takes the average of all previous records. One important note here is that, in contrast to the hospital’s approach, we used historical operations data instead of historical procedures and surgeons’ data for these methods.

The prediction models were developed and implemented in Python [42] software using stat models available in the Scikit-learn library [43] and the data manipulation and visualisation were implemented in R software [44].

We investigated the models’ performance under different scenarios. First, we compared and analysed the results of the prediction models on the test data. Then, we explored the performance of the models for the single-procedure operations (i.e., operations with only one procedure) compared to the multiple-procedure operations to further examine the models in terms of accuracy. Next, the impact of explanatory variables and the Box-Cox technique applied to the response variable were examined. Finally, the proposed ML models were compared with the current methods used in hospitals.

To examine the performance of the proposed methods, we used several error statistics including Mean Absolute Error (MAE), Median Absolute Error (MdAE), Mean Absolute Percentage Error (MAPE), and R-squared value (R²).

The MAE calculates the average magnitude of errors between the predicted values (\(F\)) by a model and the actual values (\(A\)) for \(n\) observations.where \(\sum _{i=1}^{n}|{A}_{i}- {F}_{i}|\) represents the total absolute error over \(n\) observations.

The MdAE calculates the median of all absolute differences between the predicted values by a model (F) and the actual values (A) for \(n\) observations. This metric is really useful since it is robust to outliers.

The MAPE is a statistical measure that shows the absolute error between the predicted values (F) and the actual values (A) as a percentage.

The R-squared (R²), also known as the coefficient of determination, indicates the percentage of variance in the response variable that can be explained by the features. R-squared calculates how well the given model fits the data. In the below formula, \(\stackrel{-}{A}\) represents the average of all actual values.

In "Results" section, we evaluated the performance of the ML models and showed that the XGBoost model outperformed other ML models compared, as shown in Table 3; Fig. 4. The performance of the XGBoost model was also further analysed to obtain a better understanding of any bias and predictions on subgroups within the data. While we observed more overestimation than underestimation (around 11%, see Fig. 7), the total volume of underestimation is higher, 17% to be exact. This can be attributed to the large number of records whose durations are outliers (see Fig. 8) compared to other records for which the models normally predict the operation duration to be less than the actual cases. Besides, as shown in Fig. 3, the average operation times of the specialties are varied and the performance of the model for different specialties reveals some interesting findings (see Fig. 8). From this figure, first, some specialties have a large number of outliers (i.e., the dot points outside the whiskers) including URO and GES, which are among the most common specialties for elective surgeries in the dataset. Second, for almost all specialties, the median of prediction values is considerably close, if not the same, to the median of the actual value*s. Thirdly, the prediction values for several specialties such as RAD, GAS, and GES are largely similar to the actual ones, not only in terms of median values but also first and third quartiles. Given that these are the most common specialties (as shown in Fig. 2), we can conclude that more records help the model learn better and find a better pattern for prediction. Fourthly, even for specialties with a wide range of operation duration (i.e., the boxes are long) such as CAR and GES, the box plot of the predicted values has a similar distribution shape to the actual values, which indicates the robustness of the model. There are considerable deviations between the actual and predicted values for a number of specialties, including NEP and RAT, which warrants further investigation.

In this study, we used two categories of features (i.e., explanatory variables): operation-based features and patient-based features. The effectiveness of operation-based features for predicting surgery duration is also extensively discussed in the literature [30, 32, 44, 45]. However, there are inconsistencies about the importance of patient-based features. Eijkemans et al., [30] showed that patient-based features significantly influence the performance of the models. Similarly, Bartek et al., [32] indicated that patient-based features such as age and patient class (e.g., inpatient or outpatient) are among the top features for model development. On the other hand, Edelman et al., [14] used age and the ASA score as patient-based features in their model and pointed out that these two features are not a factor in improving the model’s accuracy. Combes et al., [45] analysed different factors affecting operation durations in the endoscopy department and showed that they are not dependent on patient-based features. Our findings also showed that the developed models for surgery duration prediction substantially relied on operation-based features rather than patient-based ones (as shown in Table 5), and, based on the permutation importance technique calculations, the “specialty” was the most important feature, and the patient-based features were comparatively low importance.

Most existing research studies focus on predicting procedure-level durations as opposed to the entire operation duration (note, an operation may consist of a single or multiple procedures). However, operations include some extra workflow steps besides the underlying procedure times, which make it more challenging to be predicted. In our dataset, for operations with a single procedure, on average, 64% of the operation duration was attributed to the procedure time and the other 36% of the operation duration associated to other tasks. This ratio may differ for each procedure or specialty. The difficulty of using procedure times for estimating the entire operation time is significantly greater for operations with multiple procedures. This is because multiple procedures are not necessarily performed sequentially and may happen at the same time. The prediction models in this study yield lower MAE and MdAE but higher MAPE for single-procedure operations than multiple-procedure operations. This could be interpreted by the fact that the average and median of single-procedure operations are noticeably less than those of multiple-procedure operations, by 42 and 29 min, respectively.

As discussed in "Introduction" section, multiple models have been developed for predicting either procedure or operation durations [47]. Due to the differences in datasets, features, and experimental design, it is hard, if not impossible, to directly compare the results of the proposed models with those in the literature. However, the following presents some methodological comparisons that highlight the novelty and the effectiveness of the models developed in this study.

ShahabiKargar et al. [8] developed 3 different ML models for predicting procedure duration and found that RF obtained the best results with a MAPE of 0.68 and an R² of 0.65. Kayış et al. [16] studied two years of elective surgeries in a children’s hospital, comprising 8096 records. They used statistical methods to adjust the surgeon’s estimation for predicting the procedure time which obtained an R² of 0.69 and a MAE of 38 min. Master et al., [48] provided decision tree methods for predicting surgery time (defined as the time surgeons enter the OR to leave the OR) which yielded an R² of 0.61. They predicted the duration that surgeons rather than patients spend in the OR which is not relevant to OR utilisation. The main common limitation of these three studies is that they focus on predicting procedure times rather than the actual operation duration.

Devi et al., [19] proposed an Artificial Neural Network (ANN)-based approach for predicting operation duration. The scope of their study was limited to the operations performed within an ophthalmology department and for only three different procedures. In addition, their data was limited to only 100 records. Eijkemans et al., [30] proposed linear models for predicting operation duration using a wide range of potential features (e.g., surgical team, surgeon, and patient) which obtained an R² of 0.79. However, they focused on operations undertaken only within the general surgery department. Bodenstedt et al., [49] proposed ANN models for procedure duration prediction that achieved a MAPE of 0.37 (cf. MAPE of 0.32 for XGBoost in our study). However, their approach was limited to only laparoscopic operations. Garside et al., [50] proposed an XGBoost model for the prediction of operation duration for colorectal and spinal surgeries with an MAE of 37 min (cf. MAE of 20 min for XGBoost in our study). Edelman et al. [14] developed multiple models for predicting the total surgical procedure time (defined as total surgical time and anaesthesia time) and reported that the best prediction model estimated the actual values with an MAE of 31 min. They used the estimated surgery time as a feature that was obtained by simply subtracting 20 min from the actual total procedure time. Tuwatananurak et al., [51] developed an ML model for predicting operation duration that achieved a MdAE of 20 min (cf. MdAE of 11 min for XGBoost in our study). They collected only three months of data for model development and also used features that may not be available several days before the surgery, i.e., at the time of scheduling (e.g., cancellation of prior operations, and the time that operations started). Bartek et al., [32] developed an ML model for predicting operation duration that achieved an R² of 0.78 and a MAPE of 28%. However, they trained the data on only 12 selected specialties. A common limitation across the above-mentioned studies was that they focused only on a limited number of specialties or procedures.

While not directly comparable, our approach for operation duration prediction resulted in better performance metrics than the ones in the above-mentioned related work. In addition, we also addressed some of the limitations that exist in the literature; Firstly, we predicted the operation duration, not the procedure duration or surgery duration (i.e., the time between the surgeon entering the room and the surgeon leaving [48]). Secondly, we used a dataset collected over four years, including over 50,000 elective operations. Having a large dataset allows us to provide a comprehensive investigation of potential factors affecting the ML models for the operation duration prediction task. Thirdly, we did not restrict our models to a single or selected subset of specialties, our dataset includes over 26 specialties and 500 procedure combinations. This helps the clinical administrative team, in reality, manage ORs with a broader range of specialties. Fourthly, we used features that are all available preoperatively, i.e., several days before the surgery, which enables our approach to be deployed in the scheduling environment. Finally, our approach was able to consistently predict operation duration for both single-procedure operations as well as the more challenging multiple-procedure operations.

Comparing the XGBoost model and the MeanAll approach to actual operation duration indicated that the XGBoost model achieved 6854 min less of total absolute error over 1 year. It may be argued that the amounts gained in absolute minutes can be considered not significant, however, this can be translated to meaningful gains for the hospital and OR management in the long-term horizon. Knowing the fact that the average cost for 1 min of OR time is estimated to be between 22 and 133 USD [52], this improvement may translate to a significant reduction of costs.

The data used in this study was collected from an administrative database providing information about elective and emergency surgeries. The data used in this study did not contain surgeon-related information, including surgeon names, surgeon’s estimation of the operations and the number of anaesthetists and nurses involved in the operations. This missing information limited both our model development and our comparisons with the current hospital methods. Also, being a single-site study, our data was obtained from one hospital, and hence, was limited to a particular demographic area, a distinct pattern of surgical demand, and elective case-booking techniques. Considering the mentioned limitations, it should be noted that the broad generalisation and application of our findings should be done with caution to non-elective surgeries or other hospitals.