Implementing structured team debriefing using a Black Box in the operating room: surveying team satisfaction

In total, 35 surgical procedures were recorded of which 18 were laparoscopic fundoplications, 6 laparoscopic diaphragmatic hernia repairs, 3 elective laparoscopic appendectomies, 3 laparoscopic subtotal colectomies, 2 laparoscopic unilateral adrenalectomies, 2 laparoscopic bilateral adrenalectomies and 1 laparoscopic sigmoid resection. In these cases, 4 surgeons, 2 surgical fellows, 12 surgical residents, 6 anaesthesiologists, 5 anaesthesiology residents, 9 anaesthesiology nurses, 27 theatre nurses and 16 medical interns participated (N = 81). The baseline characteristics of participants are presented in Table 1.

The debriefings took place approximately 14 working days (median, IQR 41) after the recorded procedure. On average, 4 (out of 7–8) team members (median, IQR 3) attended their team debriefing.

In total, 151 questionnaires were completed. The mean score on the question: “How important do you feel it is to be able to structurally debrief surgical procedures with the entire team” was 8.44 (SD 1.2, 10-point Likert scale).

This study focuses on the satisfaction of the OR team with the use of a new monitoring system, the OR Black Box®, and its subsequent output used in team debriefing. This outcome was chosen because for people working in the OR it is vital to feel comfortable and secure, in order to be able to adopt such an innovative system. The team has to be satisfied with a system that ‘watches’ and ‘judges’ them. Only then, a quest to learn from unnoticed or differently perceived errors may take place [32]. Overall, satisfaction of the surgical team with the use of the OR Black Box® and corresponding outcome performance report for postoperative structured team debriefing was very high. Ninety-eight percent of participants would recommend postoperative multidisciplinary debriefing with the use of the OR Black Box® derived output to their colleagues. Although team debriefing is not yet common practice in most hospitals, participating surgical team members have considered structured team debriefing to be important, useful, and educational [17, 33‐37]. These results show that number of previously attended team debriefings is positively associated with user satisfaction. This implicates that there is no ‘wear out’ of participating to debriefing, in contrast. One may even argue that new users over time become bigger advocates for the debriefing, using the system for this purpose. The type of procedure, years working at the hospital and age did not seem to influence satisfaction, suggesting that there is no extinguish of participation interest and that bias due to the ‘novelty effect’ is minimal [38]. This is an encouraging finding, when implementing innovations in the operating theatre [39, 40].

As to be expected, the primary surgeons, drivers of the initiative, were significantly more satisfied than the participating assisting surgeon, anaesthesiologist and OR nurses in the surgical team. The phenomenon of perceived difference of perception about the same situation between the surgeon and other team members is acknowledged in literature [41‐43]. It may also be contributed to the so-called ‘Rashomon’ effect, which occurs when the same events is described in significantly different ways by different people who were involved [44]. Indeed, based on the respective roles, disagreements exist regarding the evidence of events in the OR. Also, subjectivity versus objectivity in perception, memory and reporting is in play, when looking back upon situations. Surgeons, in comparison with the other team members, experience and therefore describe or remember certain events differently. The need for a more multidisciplinary approach to quality improvement initiatives may hence be recommended [37, 45, 46]. Moreover, it is known that communication and the performance of the team is usually graded higher by the surgeon [47‐49]. This may further be explained by the fact that this project was an initiative led and strongly supported by the participating surgeons. As participants were asked to voluntarily participate in the TOPPER-trial, it was to be expected that they would be satisfied with the outcomes of project, introducing a positive selection bias in our study. Yet, at the start of the project, only a few anaesthesiologists and nurses felt comfortable enough to decide to participate and sign the informed consent. Interestingly, over time, their participation numbers kept on growing steadily in the study. An effect that can presumably be contributed to the ‘grapevine’, e.g. the positive responses of the already participating team members. Indeed, several healthcare professionals who were initially unsure or even quite sceptical towards the initiative decided to participate in the team debriefing during the trial based on positive experiences shared by their peers. When these second group of adopters overcame their initial scepticism, they reverted their opinion due to actual user experience. They came to better understand how their privacy was protected and experienced the benefits first-hand. As a result, initial laggards became the most important drivers and advocates for the initiative.

In this study, only 3 participants indicated not to recommend participation to peers, of which 1 surgical resident and 2 anaesthesiologists. The surgical resident commented that the answer was ‘no’, because during that particular debriefing, the staff surgeon had to cancel his or her attendance to the team debriefing last minute. Without the staff surgeon, in combination with a relatively ‘uneventful case’, the surgical resident considered the team debriefing to be not so useful. Two anaesthesiologists answered ‘no’ on the question if they would recommend use of the system for team debriefing to peers. Anaesthesiology data were indeed captured in real time by OR Black Box® (e.g. blood pressure, heart rate, oxygenation, etcetera) and reflected in Black Box® output, but the assessment algorithms at that time were not well enough developed to provide the same granularity of assessment as for the surgeons and OR nurses. Also, to protect the privacy of the patient, the OR Black Box® capture of data started when the team started draping, when the patient was hence already under anaesthesia. Recordings were stopped before extubation. Thus, the assumed-to-be more critical moments in anaesthesiology care were not part of the performance report and could not be debriefed using the outcome report. Nevertheless, technical aspects were not the main learning points according to user insights from both surgeons and anaesthesiologists. Take-home-messages, noted during the team debriefing sessions from the anaesthesiologists, were mainly about communication patterns, such as “clear and closed-loop communication is important” and “I should be more specific when asking the surgeon”. In fact, miscommunication has been implicated as one of the major causes of error and adverse outcomes in general surgery [10, 11]. Indeed, these learning aspects need to be taking into account when training surgical teams, which is usually not the case in the separate specialist curricula to date. Authors feel there is an opportunity here for improvement. Apart from training teams in simulative settings jointly, use of the OR Black Box® in team debriefing to look back upon joint performance may help strengthening the surgical safety culture. This, because the OR Black Box® performance report has been built focusing on those aspects regarded to be especially important for joint performance; being human factor skills, like communication and teamwork, next to technical error [50]. Postoperative multidisciplinary debriefing, with the use of the performance report, may hence contribute to prevention of unintentional miscommunication in the OR, especially between the surgeons and anaesthesiologists [51].

Taking into account the different busy work schedules and irregular shifts, planning the team debriefing sessions was difficult sometimes. However, the number of working days between procedure and debriefing session, and number of attending team members did not seem to affect the participant’s satisfaction. Nevertheless, it was decided to reschedule the session, when not enough team members could attend (4 out of 7) to persevere the benefits of multiple viewpoints in the discussion.

Several team members quoted; “because of the Black Box, I was more aware of my communication and this actually improved my way of communicating”. Yet, the performance report showed that there was still some “irrelevant chatting” or “loud music”. This indicates that procedures were performed in the familiar and natural way [26, 52, 53]. Quotes during the debriefings confirmed that there was often a very relaxed and good atmosphere in the OR. This may suggest that surveillance awareness and language did not seem to affect the surgical team’s performance and satisfaction [54].

This is not the first study describing the use of a video and MDR in the operating theatre [55‐57]. However, the TOPPER- trial is, to the author’s knowledge, the first study that used a structured and automatically analysed video-assisted performance report as a tool for structured multidisciplinary debriefing, including all members of the operating team. In contrary to others, this study comprehensively explored the participant’s satisfaction with the use of an MDR in the OR, its performance report, and debrief methods. As stated in the literature review by Jue et al., the OR Black Box® is currently the most widespread surgical data recording technology in use in operative settings [57].

This pilot study has some limitations. As mentioned, the participants were asked to voluntarily participate and therefore the results may represent the opinion of beforehand enthusiastic, positively minded participants. One out of the six participating surgeons (MS) was beside a participant, also the project leader. This is an important bias to take into account whilst interpreting the results. To avoid bias, the 6 surgeons did not participate in the data analysis. On the other hand, leading by example is not necessarily wrong in starting disruptive initiatives. One may even argue that such an initiative simply needs a strong driver from within the community and leadership in order to succeed and result in successful implementation. Overall, the level of satisfaction among various users is very high, one may argue that the system lives up to different expectations indeed and certainly did not disappoint.

Another barrier to interpretation of the study may be the fact that participants were asked to speak English during the OR Black Box® recordings. As mentioned above, the data analysis centre is situated in Toronto, Canada, and neither the software nor the ‘raters’ were able to understand and analyse Dutch. To facilitate interpretation of this learning system and maximize the information in the newly designed outcome report, authors chose upfront to revert away from bias that may have been caused by language issues. Indeed, it was believed to be not so much of a problem as the Dutch, especially when highly educated, are fluent in speaking and understanding English [58, 59]. Although it was agreed that during the procedure the team members could always revert back to Dutch if considered necessary, having to speak English was mentioned to be a limitation to the natural workflow in the evaluation of the study, especially by the OR nurses. Another limitation of the study is that its results may have been influenced by the Hawthorne effect, a well-described phenomenon of an unintentional change of behaviour or productivity in response to the presence of an ‘observer’ [60, 61]. It is known that this effect typically fades with time, as the team members are getting used to the observation, especially if the presence of an observer is not directly visible [62]. Our video recordings were made with surveillance cameras that were already mounted into the ceiling in most of our operating rooms. This non-obstructive set-up for observation is likely not to attribute much to a possible Hawthorne effect, as one is likely to forget a camera that is not disturbing one’s activities when focusing at tasks.

The patient itself was not the main subject of this study. Therefore, no correlations could be made with the operative patient outcomes or clinical endpoints. Future studies may prove the direct or indirect benefits for the patients.

Scheduling the multidisciplinary debriefings for such an amount of consecutive surgical cases with so many different team members proved to be a challenge during this study. Authors would have preferred scheduling the debriefings sooner to the surgery, but this proved not feasible in all cases. Nevertheless, having the objective information including the video footage in the outcome report sparked the memory satisfactory, according to participants. Results of this study show that neither the number of team members attending the team debriefing, nor number of workdays between the procedure and debriefing was significantly associated with the satisfaction scores. As a recommendation, authors believe that inviting OR personnel to participate in about 2 multidisciplinary debriefings per year may already be a great facilitator in better understanding each other’s need. Whether or not it is widely generalizable to have an independent person, such as a professor of psychiatry, moderate the sessions and the cost-effectiveness remains open to discussion.

As a result of the positive outcomes of this study, the OR Black Box® system is about to be implemented in full operational modus on multiple clinical operating theatres in our academic medical centre. The performance report is currently, with the help of machine learning software, continuously improving and can now be used for multiple purposes including open surgery in multiple medical centres [63]. Future studies have to determine the effect of including the recording of the entire procedure (start when patient enters the OR and stop when patient leaves the OR) and subsequent anaesthesiology data analysis feedback embedded in the performance report. Further building and incorporating deep-learning artificial intelligence software algorithms capable to process OR Black Box® data are going to provide more accurate assessment of false/true negative/positive events [64]. This may result in scalability of the model, feasibility of team debriefing and an even higher level of user satisfaction. A multicentre study is to be advocated to assess if the OR Black Box® performance report in combination with the Black Box Debrief Model is culturally robust and able to guide discussion during postoperative multidisciplinary debriefings in other medical centres as well.