nach oben

International Journal of Computer Assisted Radiology and Surgery

Erschienen in:

27.02.2024 | Original Article

Neural radiance fields-based multi-view endoscopic scene reconstruction for surgical simulation

verfasst von: Zhibao Qin, Kai Qian, Shaojun Liang, Qinhong Zheng, Jun Peng, Yonghang Tai

Erschienen in: International Journal of Computer Assisted Radiology and Surgery | Ausgabe 5/2024

Einloggen, um Zugang zu erhalten

Abstract

Purpose

In virtual surgery, the appearance of 3D models constructed from CT images lacks realism, leading to potential misunderstandings among residents. Therefore, it is crucial to reconstruct realistic endoscopic scene using multi-view images captured by an endoscope.

Methods

We propose an Endoscope-NeRF network for implicit radiance fields reconstruction of endoscopic scene under non-fixed light source, and synthesize novel views using volume rendering. Endoscope-NeRF network with multiple MLP networks and a ray transformer network represents endoscopic scene as implicit field function with color and volume density at continuous 5D vectors (3D position and 2D direction). The final synthesized image is obtained by aggregating all sampling points on each ray of the target camera using volume rendering. Our method considers the effect of distance from the light source to the sampling point on the scene radiance.

Results

Our network is validated on the lung, liver, kidney and heart of pig collected by our device. The results show that the novel views of endoscopic scene synthesized by our method outperform existing methods (NeRF and IBRNet) in terms of PSNR, SSIM, and LPIPS metrics.

Conclusion

Our network can effectively learn a radiance field function with generalization ability. Fine-tuning the pre-trained model on a new endoscopic scene to further optimize the neural radiance fields of the scene, which can provide more realistic, high-resolution rendered images for surgical simulation.

Nur mit Berechtigung zugänglich

Jensen K, Bjerrum F, Hansen HJ, Petersen RH, Pedersen JH, Konge L (2017) Using virtual reality simulation to assess competence in video-assisted thoracoscopic surgery (vats) lobectomy. Surg Endosc 31(6):2520–2528CrossRefPubMed

Chan S, Shum H-Y, Ng K-T (2007) Image-based rendering and synthesis. IEEE Signal Process Mag 24(6):22–33CrossRef

Mildenhall B, Srinivasan PP, Tancik M, Barron JT, Ramamoorthi R, Ng R (2021) Nerf: representing scenes as neural radiance fields for view synthesis. Commun ACM 65(1):99–106CrossRef

Zhang K, Riegler G, Snavely N, Koltun VN (2020) Analyzing and improving neural radiance fields. Adv Neural Inf Process Syst

Chen A, Xu Z, Zhao F, Zhang X, Xiang F, Yu J, Su H (2021) Mvsnerf: fast generalizable radiance field reconstruction from multi-view stereo. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 14124–14133

Barron JT, Mildenhall B, Verbin D, Srinivasan PP, Hedman P (2022) Mip-nerf 360: unbounded anti-aliased neural radiance fields. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5470–5479

Müller T, Evans A, Schied C, Keller A (2022) Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans Gr 41(4):1–15CrossRef

Wang Q, Wang Z, Genova K, Srinivasan PP, Zhou H, Barron JT, Martin R, Snavely N, Funkhouser T (2021) Ibrnet: learning multi-view image-based rendering. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 4690–4699

Garbin SJ, Kowalski M, Johnson M, Shotton J, Valentin J (2021) Fastnerf: high-fidelity neural rendering at 200fps. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 14346–14355

10.

Yu A, Ye V, Tancik M, Kanazawa A (2021) pixelnerf: neural radiance fields from one or few images. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 4578–4587

11.

Xu Q, Xu Z, Philip J, Bi S, Shu Z, Sunkavalli K, Neumann U (2022) Point-nerf: point-based neural radiance fields. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5438–5448

12.

Drebin RA, Carpenter L, Hanrahan P (1988) Volume rendering. ACM Siggraph Comput Gr 22(4):65–74CrossRef

13.

Levoy M, Hanrahan P (1996) Light field rendering. In: Proceedings of the 23rd annual conference on computer graphics and interactive techniques, pp 31–42

14.

Penner E, Zhang L (2017) Soft 3d reconstruction for view synthesis. ACM Trans Gr 36(6):1–11CrossRef

15.

Chlubna T, Milet T, Zemčík P (2021) Real-time per-pixel focusing method for light field rendering. Comput Vis Med 7(3):319–333CrossRef

16.

Yao Y, Luo Z, Li S, Fang T, Quan L (2018) Mvsnet: depth inference for unstructured multi-view stereo. In: Proceedings of the European conference on computer vision (ECCV), pp 767–783

17.

Zhang K, Luan F, Li Z, Snavely N (2022) Iron: inverse rendering by optimizing neural sdfs and materials from photometric images. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 5565–5574

18.

Xiangli Y, Xu L, Pan X, Zhao N, Rao A, Theobalt C, Dai B, Lin D (2022) Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering. In: European conference on computer vision. Springer, pp 106–122

19.

Kajiya JT (1986) The rendering equation. In: Proceedings of the 13th annual conference on computer graphics and interactive techniques, pp 143–150

20.

Moreno I, Viveros-Méndez P (2021) Modeling the irradiation pattern of leds at short distances. Opt Express 29(5):6845–6853CrossRefPubMed

21.

Zhu J, Zhao S, Xu Y, Meng X, Wang L, Yan L-Q (2022) Recent advances in glinty appearance rendering. Comput Vis Med, pp 1–18

22.

Mildenhall B, Srinivasan PP, Ortiz-Cayon R, Kalantari NK, Ramamoorthi R, Ng R, Kar A (2019) Local light field fusion: practical view synthesis with prescriptive sampling guidelines. ACM Trans Gr 38(4):1–14CrossRef

23.

Schonberger JL, Frahm J-M (2016) Structure-from-motion revisited. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4104–4113

24.

Dabov K, Foi A, Katkovnik V, Egiazarian K (2007) Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Trans Image Process 16(8):2080–2095CrossRefPubMed

25.

Oktay O, Schlemper J, Folgoc LL, Lee M, Heinrich M, Misawa K, Mori K, McDonagh S, Hammerla NY, Kainz B, Glocker B, Rueckert D (2018) Attention u-net: learning where to look for the pancreas. In: Medical imaging with deep learning

26.

He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

27.

Hirschberg J, Manning CD (2015) Advances in natural language processing. Science 349(6245):261–266CrossRefPubMed

28.

Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 30

29.

Kingma DP, Ba J (2014) Adam: a method for stochastic optimization. CoRR arXiv:1412.6980

Titel: Neural radiance fields-based multi-view endoscopic scene reconstruction for surgical simulation
verfasst von: Zhibao Qin
Kai Qian
Shaojun Liang
Qinhong Zheng
Jun Peng
Yonghang Tai
Publikationsdatum: 27.02.2024
Verlag: Springer International Publishing
Erschienen in: International Journal of Computer Assisted Radiology and Surgery / Ausgabe 5/2024
Print ISSN: 1861-6410
Elektronische ISSN: 1861-6429
DOI: https://doi.org/10.1007/s11548-024-03080-8

DeepPyramid+: medical image segmentation using Pyramid View Fusion and Deformable Pyramid Reception

Open Access
Original Article

Development and usability of a virtual reality umbilical venous catheter placement simulator

Original Article

Image-guided surgical planning of percutaneous nephrolithotomy with patient-specific CTRs

Original Article

PELE scores: pelvic X-ray landmark detection with pelvis extraction and enhancement

Original Article

Efficient EndoNeRF reconstruction and its application for data-driven surgical simulation

Open Access
Original Article

Subtracting–adding strategy for necrotic lesion segmentation in osteonecrosis of the femoral head

Original Article

Neu im Fachgebiet Radiologie

Stumme Schlaganfälle − ein häufiger Nebenbefund im Kopf-CT?

12.01.2025
Computertomografie
Nachrichten

In 4% der in der Notfallambulanz initiierten zerebralen Bildgebung sind „alte“ Schlaganfälle zu erkennen. Gar nicht so selten handelt es sich laut einer aktuellen Studie dabei um unbemerkte Insulte. Bietet sich hier womöglich die Chance auf ein effektives opportunistisches Screening?

Stören weiße Wände und viel Licht die Bildqualitätskontrolle?

09.01.2025
Mammografie
Nachrichten

Wenn es darum geht, die technische Qualität eines Mammogramms zu beurteilen, könnten graue Wandfarbe und reduzierte Beleuchtung im Bildgebungsraum von Vorteil sein. Darauf deuten zumindest Ergebnisse einer kleinen Studie hin.

PMBCL mit CMR: Radiatio kann ohne Risiko weggelassen werden

08.01.2025
Non-Hodgkin-Lymphome
Nachrichten

Patienten mit primär mediastinalem B-Zell-Lymphom (PMBCL), die nach der Induktionstherapie eine komplette metabolische Remission (CMR) erreichen und keine konsolidierende Bestrahlung erhalten, müssen offenbar keine Überlebensnachteile fürchten.

Das alles ändert sich für Arztpraxen in 2025

31.12.2024
EBM
Nachrichten

Ab Januar greifen jede Menge Neuerungen – ob im EBM, Arbeits- und Steuerrecht oder bei der digitalen Vernetzung. Darunter einige Verbesserungen, aber teils auch Mehraufwand. „Same procedure as every year?“ – Ein Überblick.

Update Radiologie

Bestellen Sie unseren Fach-Newsletter und bleiben Sie gut informiert.

Newsletter bestellen

Springer Medizin

Neural radiance fields-based multi-view endoscopic scene reconstruction for surgical simulation