An analysis on the effect of body tissues and surgical tools on workflow recognition in first person surgical videos

We used surgical videos recorded in breast cancer surgery by wearable cameras, Tobii Glasses2 and Tobii Glasses3 [15]. Tobii Glasses were designed as a pair of eyeglasses equipped with a scene camera and eye tracking sensors. The scene camera recorded first-person-view videos at a rate of 25 frames per second (fps). We collected nine first-person-view videos from the viewpoint of the surgeon. The duration of each video ranged between one and two hours. We attempted to record the surgery from the beginning to the end.

The full dataset used in this study was filmed after obtaining written informed consent, based on a protocol approved by the Ethics Committee of the Keio University School of Medicine (20180026).

In breast cancer surgery, distinct surgical tools were used for each workflow and three body tissues were observed (dermis, fat, and muscle). To train a machine learning model to detect body tissues and surgical tools, we annotated 400 images for body tissue segmentation and, 508 images for surgical tools segmentation. We used VGG Image Annotator [16] to label all regions. The regions were represented as polygons created by connecting dots outlining the contours. If the region of a single item was divided by an obstacle such as a hand or another item, each region was labeled as a separate region. Table 1 indicates the number of annotated images and instances. We divided the dataset for train, validation and test. In the case of body tissues, the ratio of train : validation : test was 14 : 6 : 1, while for surgical tools, the rate was 8 : 2 : 1. The dataset were split randomly.

We labeled the videos as six class labels using ELAN [17] annotation tool. The classification was instructed by a breast surgeon. The workflow classes were Before Incision (BI), Making Flap (F), Mammary Gland Dissection and Lymph Node (ML), Irrigation and Drain Insertion (ID), Skin Suture (SS) and Other Behavior (OB). Most surgeries proceeded in the aforementioned order. The OB class included the scene where the surgeon did not look at the operative field. However, if the surgeon glanced out of the operative field and then immediately returned to the operative field, the scene was not included in the OB class. The BI class included marking with a pen and Bosmin injection. Table 2 indicates the number of images used for training, validating, and testing each workflow. It is important to note that the datasets for segmentation and workflow recognition were distinct and separate entities.

We used Mask R-CNN [6] as our framework for body tissues and surgical tools segmentation. In the training of surgical tools detection, we utilized pre-training models to obtain features closely related to the domain of surgical tools. We compared three pre-trained models with datasets: ImageNet [18], EndoVis2017 dataset [19] and both of them. For EndoVis2017, we trained the Mask R-CNN model with the dataset for 35 epochs from scratch. For ImageNet, we employed the Pytorch pre-trained parameter for the Mask R-CNN model with the dataset. For ImageNet + EndoVis2017, initially, we loaded the PyTorch pre-trained parameter with ImageNet, then the model was trained with EndoVis2017 to obtain surgical tool’s tips features for 44 epochs. After that, all three pre-trained models were fine-tuned with our dataset. For the assessment of the pre-training and fine-tuning models, each model was trained five times for fivefold cross-validation of training and validation datasets, and the evaluation scores were obtained as the average of each five testing.

On the other hand, in body tissues training, we did not use any pre-trained weights, because the accuracy of segmentation was worse when we applied a pre-trained model. One of the reasons for worse was considered that the images used for pre-train were different from the surgical images.

As for other parameters, we used Adam optimizer and learning rate was \(1.0 \times 10^{-4}\). In training, data augmentations were applied, such as horizontal flip, random crop, brightness changing, and Gaussian blurring. Each image processing technique was applied with a 50 % probability. The kernel size of the Gaussian filter was \(5 \times 5\). To adjust the brightness, we randomly selected a value for \(\alpha \) between 0.5 and 1.5, and a value for \(\beta \) between -50 and 50. Any pixel values that fell below 0 or above 255 were rounded to 0 or 255, respectively. In first-person-view surgical videos, changes in brightness and motion blur occurred suddenly. These augmentations contributed to detecting regions even when the camera images were unclear.

IoU and mean Average Precision (mAP) were calculated to evaluate the segmentation accuracy with a test dataset independent of the training dataset. Table 3 and Table 4 indicate each result of segmentation. In body tissues, fat and muscle are higher than dermis because their regions are larger than dermis and they have a large number of dataset. For surgical tools, thinner tools and tools with similar shapes to others have lower detection accuracy. However, by utilizing ImageNet + EndoVis2017 for pre-training, the overall IoU and mAP metrics surpassed the individual pre-training models. Figure 2 shows snapshots of the body tissues and tool segmentation.

The top row indicates input surgical images. From left to right: marking with pen, bosmin injection, making flap, mammary gland dissection, and skin suture. The remaining rows illustrate the output of segmentation.

In body tissues segmentation, accuracy was significantly influenced by data imbalance. Fat and muscle were seen in most of the scenes, and their region sizes were larger than dermis. In comparison with that dermis was only seen the scene of beginning to open the breast. When we attempted to evenly collect training images from each scene, the number of body tissues included in surgical images became imbalanced. Consequently, the accuracy of fat and muscle detection was higher than dermis.

Identifying surgical tools can be challenging due to their thin regions and small number of pixels. Additionally, some tools, such as forceps and needle-holders, can be similar in appearance, leading to low identification accuracy. Brightness and reflection are also factors of low detection accuracy. In particular, metallic luster surgical tools sometimes showed blown out highlights areas and unclear boundaries under the light. The utilization of ImageNet as the pre-trained dataset resulted in notably lower accuracy for certain surgical tools, such as hook, forceps, tweezers, and needle-holders, in comparison with pen, syringe, scalpel, and electrical-scalpel. This discrepancy is attributed to the fact that ImageNet primarily consists of general objects, and the plastic components of surgical tools exhibit features that are more analogous to those present in the pre-trained data. Forceps are less frequently used compared to other surgical tools, and instances of forceps overlapping with each other are common. Therefore, under all of situations, it is presumed that the accuracy of instance segmentation is lower compared to other surgical tools.

The network of workflow classification was Trans-SVNet [20]. In Trans-SVNet, ResNet [21] was trained for spatial embedding, and temporal convolutional network(TCN) was trained for temporal embedding. A sequence of temporal embedding was fed into one of the Transformer layers. Then, another Transformer was trained to use these two embeddings as spatial-temporal hybrid embedding. The network’s output consisted of the probability of each workflow in the range [0, 1], determined through the application of softmax, and the backward propagation of the cross-entropy loss. In this study, three different inputs were compared to verify identification accuracy. The first input consisted solely of RGB images, which is baseline(RGB). The second input consisted RGB images with three-channel body tissue masks (RGB+BT). The third input consisted RGB images with three-channel body tissues and eight-channel surgical tools masks (RGB+BT+ST).

Seven complete surgical videos were used in the training. Five of them were used for the training model, one was used for validation, and the remaining videos were used for testing. All surgical videos were recorded at 25 fps and were downsampled to 1 fps. For temporal embedding, the sequence size was set to 30 s. In Trans-SVNet, data augmentations were applied, including horizontal flipping, random cropping, random rotation, and color adjustments. When the input was RGB surgical images and region masks, color jitter was applied only RGB surgical images not region masks, and other data augmentations were applied to both types of data.

Figure 3 indicates an example of the transition of body tissues region sizes. The region sizes of body tissues were expressed as percentages relative to the full image pixels. In the first half of the surgery, there were mainly fat and dermis regions, and muscle regions appeared in the midfield. As the incision progressed, the region size of fat and muscle increased. During the Skin Suture phase, the size of these regions rapidly decreased. When the surgeon did not focus on the operative field, the values were 0. The observed transition pattern in the sizes of body tissues in breast surgery videos appears to be consistent across the recorded videos.

Table 5 shows the accuracy of workflow recognition using Trans-SVNet. Precision, recall, and F1 were calculated for each workflow. The higher value in each workflow is highlighted in bold. Overall, the addition of body tissues and surgical tools masks resulted in higher precision and recall compared to using only RGB images. Specifically, the RGB+BT method achieved nearly 3 % higher precision than the baseline, while the RGB+BT+ST method achieved 5 % higher precision. Additionally, both region-aware methods achieved 4 % higher recall than the baseline method. Similarly for F1, the result was improved by 3 % when the input was RGB+BT, and RGB+BT+ST was nearly 5 % higher than the baseline. Figure 4 compares the predicted workflow among the three different inputs. Although each input has some short-term incorrect predictions, the figure indicates RGB+BT showed fewer errors than other models.

Between 15 and 30 min, all prediction models misidentified the Making Flap as the Mammary Gland Dissection and Lymph Node due to their similar appearance and the intermittent nature of these workflows. In Making Flap phase, there were a large number of prediction errors, resulting in lower precision and recall. Furthermore, during the Other Behavior phase between 100 min and 120 min, it was mistakenly identified as the Making Flap and other workflows. In this scene, the surgeons held around the sutured skin with their hands, and it was similar to the scene at the beginning of the skin incision. At the last 20 min of the surgery, the models of RGB+BT and RGB+BT+ST estimated the workflow correctly despite of errors in the body tissue and surgical tool segmentation.

Figure 4 indicates that the short-time errors in the RGB model were reduced in the RGB+BT and RGB+BT+ST models by considering the transition of body tissues. For RQ1, we found that tracking body tissues region was indeed a valid factor to improve the accuracy of procedural workflow recognition. For RQ2, the result suggests that both body tissues and surgical tools contribute to the improvement of procedural workflow recognition accuracy. We revealed that explicitly representing the state of the operative field leads to a reduction in ambiguity and improved precision in procedural workflow recognition. Nevertheless, challenges persist in achieving accurate segmentation. However, there is potential for future improvements in accuracy through advancements in the methodology.

This study has demonstrated the potential for further development in video-based surgical education for open surgery. Jason L. Green et al. [22] revealed that video-based education, supplementing educational tools such as texts, demonstrated statistically significant increases in anatomical and procedural knowledge, as well as improvement in surgical performance. The findings in this study contributes to making educational video contents by extracting relevant video clipping and creating chapters in consideration of body tissues and surgical tools.