Skip to main content
Erschienen in: International Journal of Computer Assisted Radiology and Surgery 4/2019

18.02.2019 | Original Article

Automatic and near real-time stylistic behavior assessment in robotic surgery

verfasst von: M. Ershad, R. Rege, Ann Majewicz Fey

Erschienen in: International Journal of Computer Assisted Radiology and Surgery | Ausgabe 4/2019

Einloggen, um Zugang zu erhalten

Abstract

Purpose

Automatic skill evaluation is of great importance in surgical robotic training. Extensive research has been done to evaluate surgical skill, and a variety of quantitative metrics have been proposed. However, these methods primarily use expert selected features which may not capture latent information in movement data. In addition, these features are calculated over the entire task time and are provided to the user after the completion of the task. Thus, these quantitative metrics do not provide users with information on how to modify their movements to improve performance in real time. This study focuses on automatic stylistic behavior recognition that has the potential to be implemented in near real time.

Methods

We propose a sparse coding framework for automatic stylistic behavior recognition in short time intervals using only position data from the hands, wrist, elbow, and shoulder. A codebook is built for each stylistic adjective using the positive and negative labels provided for each trial through crowd sourcing. Sparse code coefficients are obtained for short time intervals (0.25 s) in a trial using this codebook. A support vector machine classifier is trained and validated through tenfold cross-validation using the sparse codes from the training set.

Results

The results indicate that the proposed dictionary learning method is able to assess stylistic behavior performance in near real time using user joint position data with improved accuracy compared to using PCA features or raw data.

Conclusion

The possibility to automatically evaluate a trainee’s style of movement in short time intervals could provide the user with online customized feedback and thus improve performance during surgical tasks.
Literatur
1.
Zurück zum Zitat Aghdasi N, Bly R, White LW, Hannaford B, Moe K, Lendvay TS (2015) Crowd-sourced assessment of surgical skills in cricothyrotomy procedure. J Surg Res 196(2):302–306CrossRefPubMedPubMedCentral Aghdasi N, Bly R, White LW, Hannaford B, Moe K, Lendvay TS (2015) Crowd-sourced assessment of surgical skills in cricothyrotomy procedure. J Surg Res 196(2):302–306CrossRefPubMedPubMedCentral
2.
Zurück zum Zitat Aharon M, Elad M, Bruckstein A (2006) rmk-svd: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans Signal Process 54(11):4311–4322CrossRef Aharon M, Elad M, Bruckstein A (2006) rmk-svd: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans Signal Process 54(11):4311–4322CrossRef
3.
Zurück zum Zitat Birkmeyer JD, Finks JF, O’reilly A, Oerline M, Carlin AM, Nunn AR, Dimick J, Banerjee M, Birkmeyer NJ (2013) Surgical skill and complication rates after bariatric surgery. N Engl J Med 369(15):1434–1442CrossRefPubMed Birkmeyer JD, Finks JF, O’reilly A, Oerline M, Carlin AM, Nunn AR, Dimick J, Banerjee M, Birkmeyer NJ (2013) Surgical skill and complication rates after bariatric surgery. N Engl J Med 369(15):1434–1442CrossRefPubMed
4.
Zurück zum Zitat Chen C, White L, Kowalewski T, Aggarwal R, Lintott C, Comstock B, Kuksenok K, Aragon C, Holst D, Lendvay T (2014) Crowd-sourced assessment of technical skills: a novel method to evaluate surgical performance. J Surg Res 187(1):65–71CrossRefPubMed Chen C, White L, Kowalewski T, Aggarwal R, Lintott C, Comstock B, Kuksenok K, Aragon C, Holst D, Lendvay T (2014) Crowd-sourced assessment of technical skills: a novel method to evaluate surgical performance. J Surg Res 187(1):65–71CrossRefPubMed
5.
Zurück zum Zitat Chen SP, Kirsch S, Zlatev DV, Chang T, Comstock B, Lendvay TS, Liao JC (2016) Optical biopsy of bladder cancer using crowd-sourced assessment. J Am Med Assoc (JAMA) Surg 151(1):90–93 Chen SP, Kirsch S, Zlatev DV, Chang T, Comstock B, Lendvay TS, Liao JC (2016) Optical biopsy of bladder cancer using crowd-sourced assessment. J Am Med Assoc (JAMA) Surg 151(1):90–93
6.
Zurück zum Zitat Darzi A, Mackay S (2001) Assessment of surgical competence. BMJ Qual Saf 10(suppl 2):64–69 Darzi A, Mackay S (2001) Assessment of surgical competence. BMJ Qual Saf 10(suppl 2):64–69
7.
Zurück zum Zitat Datta V, Mackay S, Mandalia M, Darzi A (2001) The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model. J Am Coll Surg 193(5):479–485CrossRefPubMed Datta V, Mackay S, Mandalia M, Darzi A (2001) The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model. J Am Coll Surg 193(5):479–485CrossRefPubMed
8.
Zurück zum Zitat Datta V, Chang A, Mackay S, Darzi A (2002) The relationship between motion analysis and surgical technical assessments. Am J Surg 184(1):70–73CrossRefPubMed Datta V, Chang A, Mackay S, Darzi A (2002) The relationship between motion analysis and surgical technical assessments. Am J Surg 184(1):70–73CrossRefPubMed
9.
Zurück zum Zitat Deal SB, Lendvay TS, Haque MI, Brand T, Comstock B, Warren J, Alseidi A (2016) Crowd-sourced assessment of technical skills: an opportunity for improvement in the assessment of laparoscopic surgical skills. Am J Surg 211(2):398–404CrossRefPubMed Deal SB, Lendvay TS, Haque MI, Brand T, Comstock B, Warren J, Alseidi A (2016) Crowd-sourced assessment of technical skills: an opportunity for improvement in the assessment of laparoscopic surgical skills. Am J Surg 211(2):398–404CrossRefPubMed
10.
Zurück zum Zitat Derossis AM, Fried GM, Abrahamowicz M, Sigman HH, Barkun JS, Meakins JL (1998) Development of a model for training and evaluation of laparoscopic skills. Am J Surg 175(6):482–487CrossRefPubMed Derossis AM, Fried GM, Abrahamowicz M, Sigman HH, Barkun JS, Meakins JL (1998) Development of a model for training and evaluation of laparoscopic skills. Am J Surg 175(6):482–487CrossRefPubMed
11.
Zurück zum Zitat Dhamala M, Rangarajan G, Ding M (2008) Estimating granger causality from Fourier and wavelet transforms of time series data. Phys Rev Lett 100(1):018701CrossRefPubMed Dhamala M, Rangarajan G, Ding M (2008) Estimating granger causality from Fourier and wavelet transforms of time series data. Phys Rev Lett 100(1):018701CrossRefPubMed
12.
Zurück zum Zitat Efron B, Hastie T, Johnstone I, Tibshirani R (2004) Least angle regression. Ann Stat 32(2):407–499CrossRef Efron B, Hastie T, Johnstone I, Tibshirani R (2004) Least angle regression. Ann Stat 32(2):407–499CrossRef
13.
Zurück zum Zitat El Moudden I, Ouzir M, Benyacoub B, ElBernoussi S (2016) Mining human activity using dimensionality reduction and pattern recognition. Contemp Eng Sci: CES 9:21CrossRef El Moudden I, Ouzir M, Benyacoub B, ElBernoussi S (2016) Mining human activity using dimensionality reduction and pattern recognition. Contemp Eng Sci: CES 9:21CrossRef
14.
Zurück zum Zitat Ershad M, Koesters Z, Rege R, Majewicz A (2016) Meaningful assessment of surgical expertise: semantic labeling with data and crowds. In: International conference on medical image computing and computer-assisted intervention (MICCAI). Springer, pp 508–515 Ershad M, Koesters Z, Rege R, Majewicz A (2016) Meaningful assessment of surgical expertise: semantic labeling with data and crowds. In: International conference on medical image computing and computer-assisted intervention (MICCAI). Springer, pp 508–515
15.
Zurück zum Zitat Ershad M, Rege R, Fey AM (2018a) Meaningful assessment of robotic surgical style using the wisdom of crowds. Int J Comput Assist Radiol Surg: IJCARS, 1–12 Ershad M, Rege R, Fey AM (2018a) Meaningful assessment of robotic surgical style using the wisdom of crowds. Int J Comput Assist Radiol Surg: IJCARS, 1–12
16.
Zurück zum Zitat Ershad M, Rege R, Majewicz A (2018b) Surgical skill level assessment using automatic feature extraction methods. In: Medical imaging: image-guided procedures, robotic interventions, and modeling. International Society for Optics and Photonics, p 6 Ershad M, Rege R, Majewicz A (2018b) Surgical skill level assessment using automatic feature extraction methods. In: Medical imaging: image-guided procedures, robotic interventions, and modeling. International Society for Optics and Photonics, p 6
17.
Zurück zum Zitat Fard MJ, Ameri S, Darin Ellis R, Chinnam RB, Pandya AK, Klein MD (2018) Automated robot-assisted surgical skill evaluation: predictive analytics approach. Int J Med Robot Comput Assist Surg 14(1):e1850CrossRef Fard MJ, Ameri S, Darin Ellis R, Chinnam RB, Pandya AK, Klein MD (2018) Automated robot-assisted surgical skill evaluation: predictive analytics approach. Int J Med Robot Comput Assist Surg 14(1):e1850CrossRef
21.
Zurück zum Zitat Holst D, Kowalewski TM, White LW, Brand TC, Harper JD, Sorenson MD, Kirsch S, Lendvay TS (2015) Crowd-sourced assessment of technical skills: an adjunct to urology resident surgical simulation training. J Endourol 29(5):604–609CrossRefPubMed Holst D, Kowalewski TM, White LW, Brand TC, Harper JD, Sorenson MD, Kirsch S, Lendvay TS (2015) Crowd-sourced assessment of technical skills: an adjunct to urology resident surgical simulation training. J Endourol 29(5):604–609CrossRefPubMed
22.
Zurück zum Zitat Hoyer PO (2002) Non-negative sparse coding. In: Proceedings of the 12th IEEE workshop on neural networks for signal processing. IEEE, pp 557–565 Hoyer PO (2002) Non-negative sparse coding. In: Proceedings of the 12th IEEE workshop on neural networks for signal processing. IEEE, pp 557–565
23.
Zurück zum Zitat Hung AJ, Chen J, Che Z, Nilanon T, Jarc A, Titus M, Oh PJ, Gill IS, Liu Y (2018) Utilizing machine learning and automated performance metrics to evaluate robot-assisted radical prostatectomy performance and predict outcomes. J Endourol 32(5):438–444CrossRefPubMed Hung AJ, Chen J, Che Z, Nilanon T, Jarc A, Titus M, Oh PJ, Gill IS, Liu Y (2018) Utilizing machine learning and automated performance metrics to evaluate robot-assisted radical prostatectomy performance and predict outcomes. J Endourol 32(5):438–444CrossRefPubMed
24.
Zurück zum Zitat Jin A, Yeung S, Jopling J, Krause J, Azagury D, Milstein A, Fei-Fei L (2018) Tool detection and operative skill assessment in surgical videos using region-based convolutional neural networks. arXiv preprint arXiv:1802.08774 Jin A, Yeung S, Jopling J, Krause J, Azagury D, Milstein A, Fei-Fei L (2018) Tool detection and operative skill assessment in surgical videos using region-based convolutional neural networks. arXiv preprint arXiv:​1802.​08774
25.
Zurück zum Zitat Karg M, Jenke R, Seiberl W, Kühnlenz K, Schwirtz A, Buss M (2009) A comparison of pca, kpca and lda for feature extraction to recognize affect in gait kinematics. In: 3rd IEEE international conference on affective computing and intelligent interaction and workshops (ACII). IEEE, pp 1–6 Karg M, Jenke R, Seiberl W, Kühnlenz K, Schwirtz A, Buss M (2009) A comparison of pca, kpca and lda for feature extraction to recognize affect in gait kinematics. In: 3rd IEEE international conference on affective computing and intelligent interaction and workshops (ACII). IEEE, pp 1–6
26.
Zurück zum Zitat Kazanzides P, Chen Z, Deguet A, Fischer GS, Taylor RH, DiMaio SP (2014) An open-source research kit for the da vinci® surgical system. In: 2014 IEEE international conference on robotics and automation (ICRA). IEEE, pp 6434–6439 Kazanzides P, Chen Z, Deguet A, Fischer GS, Taylor RH, DiMaio SP (2014) An open-source research kit for the da vinci® surgical system. In: 2014 IEEE international conference on robotics and automation (ICRA). IEEE, pp 6434–6439
27.
Zurück zum Zitat Kirkwood RN, Resende RA, Magalhães C, Gomes HA, Mingoti SA, Sampaio RF (2011) Application of principal component analysis on gait kinematics in elderly women with knee osteoarthritis. Braz J Phys Ther 15(1):52–58CrossRef Kirkwood RN, Resende RA, Magalhães C, Gomes HA, Mingoti SA, Sampaio RF (2011) Application of principal component analysis on gait kinematics in elderly women with knee osteoarthritis. Braz J Phys Ther 15(1):52–58CrossRef
28.
Zurück zum Zitat Kowalewski TM, Comstock B, Sweet R, Schaffhausen C, Menhadji A, Averch T, Box G, Brand T, Ferrandino M, Kaouk J (2016) Crowd-sourced assessment of technical skills for validation of basic laparoscopic urologic skills tasks. J Urol 195(6):1859–1865CrossRefPubMed Kowalewski TM, Comstock B, Sweet R, Schaffhausen C, Menhadji A, Averch T, Box G, Brand T, Ferrandino M, Kaouk J (2016) Crowd-sourced assessment of technical skills for validation of basic laparoscopic urologic skills tasks. J Urol 195(6):1859–1865CrossRefPubMed
29.
Zurück zum Zitat Law H, Ghani K, Deng J (2017) Surgeon technical skill assessment using computer vision based analysis. In: Machine learning for healthcare conference, pp 88–99 Law H, Ghani K, Deng J (2017) Surgeon technical skill assessment using computer vision based analysis. In: Machine learning for healthcare conference, pp 88–99
30.
Zurück zum Zitat Liang K, Xing Y, Li J, Wang S, Li A, Li J (2018) Motion control skill assessment based on kinematic analysis of robotic end-effector movements. Int J Med Robot Comput Assist Surg 14(1):e1845CrossRef Liang K, Xing Y, Li J, Wang S, Li A, Li J (2018) Motion control skill assessment based on kinematic analysis of robotic end-effector movements. Int J Med Robot Comput Assist Surg 14(1):e1845CrossRef
31.
Zurück zum Zitat Maier-Hein L, Vedula SS, Speidel S, Navab N, Kikinis R, Park A, Eisenmann M, Feu-ssner H, Forestier G, Giannarou S, Hashizume M, Katic D, Kenngott H, Kranzfelder M, Malpani A, März K, Neumuth T, Padoy N, Pugh CM, Schoch N, Stoyanov D, Taylor R, Wagner M, Hager GD, Jannin P (2017) Surgical data science for next-generation interventions. Nat Biomed Eng 1:691–696CrossRefPubMed Maier-Hein L, Vedula SS, Speidel S, Navab N, Kikinis R, Park A, Eisenmann M, Feu-ssner H, Forestier G, Giannarou S, Hashizume M, Katic D, Kenngott H, Kranzfelder M, Malpani A, März K, Neumuth T, Padoy N, Pugh CM, Schoch N, Stoyanov D, Taylor R, Wagner M, Hager GD, Jannin P (2017) Surgical data science for next-generation interventions. Nat Biomed Eng 1:691–696CrossRefPubMed
32.
Zurück zum Zitat Mallat SG (1989) A theory for multiresolution signal decomposition: the wavelet representation. IEEE Trans Pattern Anal Mach Intell 11(7):674–693CrossRef Mallat SG (1989) A theory for multiresolution signal decomposition: the wavelet representation. IEEE Trans Pattern Anal Mach Intell 11(7):674–693CrossRef
35.
Zurück zum Zitat Milovanović I, Popović DB (2012) Principal component analysis of gait kinematics data in acute and chronic stroke patients. Comput Math Methods Med 2012:8 Milovanović I, Popović DB (2012) Principal component analysis of gait kinematics data in acute and chronic stroke patients. Comput Math Methods Med 2012:8
36.
Zurück zum Zitat Nisky I, Hsieh MH, Okamura AM (2013) A framework for analysis of surgeon arm posture variability in robot-assisted surgery. In: IEEE international conference on robotics and automation (ICRA). IEEE, pp 245–251 Nisky I, Hsieh MH, Okamura AM (2013) A framework for analysis of surgeon arm posture variability in robot-assisted surgery. In: IEEE international conference on robotics and automation (ICRA). IEEE, pp 245–251
37.
Zurück zum Zitat Nisky I, Hsieh MH, Okamura AM (2014) Uncontrolled manifold analysis of arm joint angle variability during robotic teleoperation and freehand movement of surgeons and novices. IEEE Trans Biomed Eng 61(12):2869–2881CrossRefPubMed Nisky I, Hsieh MH, Okamura AM (2014) Uncontrolled manifold analysis of arm joint angle variability during robotic teleoperation and freehand movement of surgeons and novices. IEEE Trans Biomed Eng 61(12):2869–2881CrossRefPubMed
38.
Zurück zum Zitat Polin MR, Siddiqui NY, Comstock BA, Hesham H, Brown C, Lendvay TS, Martino MA (2016) Crowdsourcing: a valid alternative to expert evaluation of robotic surgery skills. Am J Obstet Gynecol 215(5):644–e1CrossRefPubMed Polin MR, Siddiqui NY, Comstock BA, Hesham H, Brown C, Lendvay TS, Martino MA (2016) Crowdsourcing: a valid alternative to expert evaluation of robotic surgery skills. Am J Obstet Gynecol 215(5):644–e1CrossRefPubMed
39.
Zurück zum Zitat Reiley CE, Hager GD (2009) Task versus subtask surgical skill evaluation of robotic minimally invasive surgery. In: International conference on medical image computing and computer-assisted intervention (MICCAI). Springer, pp 435–442 Reiley CE, Hager GD (2009) Task versus subtask surgical skill evaluation of robotic minimally invasive surgery. In: International conference on medical image computing and computer-assisted intervention (MICCAI). Springer, pp 435–442
40.
Zurück zum Zitat Reiley CE, Lin HC, Varadarajan B, Vagvolgyi B, Khudanpur S, Yuh DD, Hager GD (2008) Automatic recognition of surgical motions using statistical modeling for capturing variability. Stud Health Technol Inform 132(1):396–401PubMed Reiley CE, Lin HC, Varadarajan B, Vagvolgyi B, Khudanpur S, Yuh DD, Hager GD (2008) Automatic recognition of surgical motions using statistical modeling for capturing variability. Stud Health Technol Inform 132(1):396–401PubMed
42.
Zurück zum Zitat Smith S, Torkington J, Brown T, Taffinder N, Darzi A (2002) Motion analysis. Surg Endosc 16(4):640–645CrossRefPubMed Smith S, Torkington J, Brown T, Taffinder N, Darzi A (2002) Motion analysis. Surg Endosc 16(4):640–645CrossRefPubMed
43.
Zurück zum Zitat Varadarajan B, Reiley C, Lin H, Khudanpur S, Hager G (2009) Data-derived models for segmentation with application to surgical assessment and training. In: International conference on medical image computing and computer-assisted intervention (MICCAI). Springer, pp 426–434 Varadarajan B, Reiley C, Lin H, Khudanpur S, Hager G (2009) Data-derived models for segmentation with application to surgical assessment and training. In: International conference on medical image computing and computer-assisted intervention (MICCAI). Springer, pp 426–434
46.
Zurück zum Zitat White LW, Kowalewski TM, Dockter RL, Comstock B, Hannaford B, Lendvay TS (2015) Crowd-sourced assessment of technical skill: a valid method for discriminating basic robotic surgery skills. J Endourol 29(11):1295–1301CrossRefPubMed White LW, Kowalewski TM, Dockter RL, Comstock B, Hannaford B, Lendvay TS (2015) Crowd-sourced assessment of technical skill: a valid method for discriminating basic robotic surgery skills. J Endourol 29(11):1295–1301CrossRefPubMed
47.
Zurück zum Zitat Wright J, Yang AY, Ganesh A, Sastry SS, Ma Y (2009) Robust face recognition via sparse representation. IEEE Trans Pattern Anal Mach Intell 31(2):210–227CrossRefPubMed Wright J, Yang AY, Ganesh A, Sastry SS, Ma Y (2009) Robust face recognition via sparse representation. IEEE Trans Pattern Anal Mach Intell 31(2):210–227CrossRefPubMed
Metadaten
Titel
Automatic and near real-time stylistic behavior assessment in robotic surgery
verfasst von
M. Ershad
R. Rege
Ann Majewicz Fey
Publikationsdatum
18.02.2019
Verlag
Springer International Publishing
Erschienen in
International Journal of Computer Assisted Radiology and Surgery / Ausgabe 4/2019
Print ISSN: 1861-6410
Elektronische ISSN: 1861-6429
DOI
https://doi.org/10.1007/s11548-019-01920-6

Weitere Artikel der Ausgabe 4/2019

International Journal of Computer Assisted Radiology and Surgery 4/2019 Zur Ausgabe

Endlich: Zi zeigt, mit welchen PVS Praxen zufrieden sind

IT für Ärzte Nachrichten

Darauf haben viele Praxen gewartet: Das Zi hat eine Liste von Praxisverwaltungssystemen veröffentlicht, die von Nutzern positiv bewertet werden. Eine gute Grundlage für wechselwillige Ärzte und Psychotherapeuten.

Akuter Schwindel: Wann lohnt sich eine MRT?

28.04.2024 Schwindel Nachrichten

Akuter Schwindel stellt oft eine diagnostische Herausforderung dar. Wie nützlich dabei eine MRT ist, hat eine Studie aus Finnland untersucht. Immerhin einer von sechs Patienten wurde mit akutem ischämischem Schlaganfall diagnostiziert.

Screening-Mammografie offenbart erhöhtes Herz-Kreislauf-Risiko

26.04.2024 Mammografie Nachrichten

Routinemäßige Mammografien helfen, Brustkrebs frühzeitig zu erkennen. Anhand der Röntgenuntersuchung lassen sich aber auch kardiovaskuläre Risikopatientinnen identifizieren. Als zuverlässiger Anhaltspunkt gilt die Verkalkung der Brustarterien.

S3-Leitlinie zu Pankreaskrebs aktualisiert

23.04.2024 Pankreaskarzinom Nachrichten

Die Empfehlungen zur Therapie des Pankreaskarzinoms wurden um zwei Off-Label-Anwendungen erweitert. Und auch im Bereich der Früherkennung gibt es Aktualisierungen.

Update Radiologie

Bestellen Sie unseren Fach-Newsletter und bleiben Sie gut informiert.