1 Introduction

Peripheral vascular access is one of the most commonly performed clinical procedures in the world and is a pivotal step in the diagnosis and treatment of many medical conditions. Oftentimes, however, it can be difficult to identify suitable blood vessels, particularly in patients with small vessels, dark skin, or a high body mass [1]. It may also be difficult to estimate the depth of the vessel or accurately insert the needle due to the lack of visibility through the skin.

A number of imaging tools have been introduced in recent years to assist clinicians in performing vascular access. Optical imaging systems most commonly utilize near infrared (NIR) light, as the decreased scattering of NIR light through tissue may allow otherwise occluded vessels up to 3 mm below the skin to be detected. Meanwhile, to visualize vessels beyond a few millimeters in depth, ultrasound (US) imaging is generally preferred. Unfortunately, when either imaging modality is used alone, trade-offs must be made between image resolution, penetration depth, and field-of-view (FOV). Furthermore, for the imaging tools to provide clinical benefit, improvements in vessel visualization must be translated into an increase in needle insertion accuracy.

To address these challenges, we have developed a portable device for peripheral vessel imaging (Fig. 1a) that uses NIR light to detect vessels over the FOV of an adult forearm; US to provide local high-resolution scans of a selected vessel target; and image analysis routines to segment and track the vessels in 3D at video rates. This paper describes the hardware and software design of the NIR+US imaging device and provides evidence of the device’s clinical potential - both as a standalone tool and as a means for image-guided robotic cannulation.

2 Methods

2.1 Bimodal NIR+US Vessel Imaging

Near Infrared Stereo Imaging: The main components of the NIR imaging subsystem are shown in Figs. 1b and c. The light source consists of six arrays of light emitting diodes (LEDs) with wavelengths of 757 and 914 nm. The wavelengths were selected to maximize absorption due to blood while minimizing absorption due to water and fat [2] (Fig. 1d). A cylindrical reflector is used to redirect the upward emitted light back toward the skin in a diffuse manner (Fig. 1b). Two miniature CMOS cameras (VRmUsb12, VRMagic UAB, DEU) form a stereo vision system that acquires 752\(\,\times \,\)480 images at 40 frames per second (fps). Each camera is coupled with a wide-angle (\(120^\circ \)) lens, a 750 nm long-pass filter, and a NIR polarizing filter oriented orthogonal to a second set of polarizers above the LED arrays. The camera parameters are computed based on geometric calibration using circular control points [3].

Fig. 1.
figure 1

Bimodal NIR+US vessel imaging. (a) The two imaging modalities are combined to provide 3D position information for needle guidance. (b) Main imaging hardware components. Red lines show the light path when using the inverted reflection imaging configuration. (c) Miniaturized optical components in each NIR camera subsystem. (d) 757 and 914 nm LEDs are used to maximize absorption from deoxyhemoglobin (DHb) and oxyhemoglobin (OHb) while minimizing absorption due to water and fat. (Color figure online)

Full size image

Ultrasound Imaging: US transducers may be easily interchanged within the system depending on resolution requirements. In this study we used a high-frequency (18 MHz) linear array transducer with 100 \(\mu \)m element pitch (L18-10L30H-4, Telemed UAB, LTU) (Fig. 1a and b). The transducer provides B-mode and Color Doppler images at up to 40 fps and with sufficient resolution to delineate vessels 1 mm in diameter or greater. Solid polyacrylamide hydrogel is used in place of liquid ultrasound gel to improve device usability.

2.2 Segmentation, 3D Reconstruction, and Motion Tracking

The image analysis routines (Fig. 2) are executed on a laptop computer equipped with a CUDA-enabled GPU. During the procedure, the clinician selects a target vessel (\(x_t\), \(y_t\), \(z_t\)) for cannulation using the graphical user interface (GUI).

Fig. 2.
figure 2

Overview of the NIR and US image analysis framework. Gray boxes indicate routines performed on the CPU. Red boxes indicate GPU-enabled computations. (Color figure online)

Full size image
Fig. 3.
figure 3

Image analysis of NIR images. (a) Segmentation of left and right forearm images, including GVF initialization (blue) and final result (red). (b–e) Edge enhancement, Hessian vessel enhancement, distance transform, and skeleton. (f) Point registration (white squares) along epipolar lines (red stripes). (g) 3D reconstruction of vessels (red) and vessel target (blue circle). (h) 3D arm surface and (i) surface normals (blue arrows). (j) Normal vector \(\mathbf {n}_t\) and tangent plane (blue square) about vessel target. (Color figure online)

Full size image

Arm Segmentation Based on Active Contours: Gradient vector flow (GVF) based active contour segmentation [4] is then performed to extract the 2D region-of-interest (ROI) of the arm in each image (Fig. 3a). We used GVF active contours for its insensitivity to initialization and ability to move into concavities. The computation is accelerated with a CUDA-optimized OpenCL implementation of GVF [5]. In the first frame, the contour is initialized by threshold and morphological operations. In all subsequent frames, the contours are initialized using the segmentation result from the preceeding frame.

Vessel Segmentation Using Curvilinear Filters: The vessels are segmented based on the assumption that they have a curvilinear line- or tube-like structure. Specifically, a Difference of Gaussians filter is first used to enhance line structures in the image (Fig. 3b). A Hessian-based filtering method [6] is then performed to enhance tubular structures (Fig. 3c).

Stereo Reconstruction: The 3D positions (\(x_i\), \(y_i\), \(z_i\)) along each vessel are computed by extracting feature points i and performing point registration between the left and right stereo images. We use the local intensity maxima of the distance map of the vessel segmentation images as feature points (Fig. 3d). The points lie along the vessel centerlines (Fig. 3e), and the intensity of each point represents the vessel diameter about i. Non-rigid point registration [7] is constrained to occur only along the epipolar lines between the stereo cameras (Fig. 3f). The resulting 3D vessel geometry is shown in Fig. 3g.

In addition to vessel reconstruction, the 3D arm surface is also extracted when the full 6-DOF pose information (\(x_i\), \(y_i\), \(z_i\), \(\alpha _i\), \(\beta _i\), \(\gamma _i\)) about each feature point i can be utilized, i.e. for robotic guidance. Here, we use a CUDA-optimized belief propagation algorithm [8] for real-time dense stereo correspondence. Figure 3h shows the resulting 3D point cloud of the arm. The surface normal vectors about each 3D point (Fig. 3i) define the pose of the vessels below (Fig. 3j).

Vessel Motion Tracking: Pairwise vessel feature points are tracked between every frame using OpenCV CUDA-optimized Lucase-Kanade pyramidal optical flow [9]. Pairs are discarded if either point deviates from the epipolar line beyond a minimum tolerance. At every fifth frame, newly registered feature points are used to update the vessel target based on proximity to the optical flow position.

Vessel Segmentation from US Images: To isolate the target vessel in the US image after the transducer is lowered over the skin, we again apply the GVF active contour model for segmentation at every second frame. As with the forearm segmentation from NIR images, the contour is initialized in the first US frame by threshold and morphological operations. The US system may also be used to confirm blood flow by displaying Color Doppler images onto the GUI during the procedure. Finally, US is useful to visualize the needle within the tissue; vessels may also roll or deflect during insertion, and this may only be detected with US. Ongoing work is focused on implementing needle segmentation and tracking approaches as an added means of safety and image feedback.

Fig. 4.
figure 4

Imaging assessment on 9 adult subjects. (a) Representative NIR segmentation and 3D reconstruction results. (b) Representative US segmentation results, with initialization shown in blue and final contours in red. (c) Average per-frame processing times. (d) Root mean squared errors relative to manual segmentation and tracking. (Color figure online)

Full size image

3 Results and Discussion

3.1 Assessment of NIR+US Vessel Imaging in 9 Adult Subjects

The speed and accuracy of each image analysis step were investigated on 9 adult subjects (Fig. 4). Representative NIR segmentation, 3D reconstruction, and US segmentation results are shown in Figs. 4a and b. Figure 4c lists the average processing times for each step. Unsurprisingly, the stereo matching routines were the most demanding (80 and 125 ms). The remaining steps each required less than 50 ms. The overall frame rate was maintained at 10.8 fps by executing independent steps in parallel and carrying out demanding computations over multiple frames.

Figure 4d shows root mean squared errors (RMSE) for the arm segmentation, NIR vessel segmentation, vessel tracking, and US vessel segmentation steps over 30 s relative to manual segmentation and tracking. Errors are expressed as a % of the average per-frame segmentation area or motion displacement. The greatest error (9.6 %) was observed in the NIR vessel segmentation step, largely due to the removal of small disconnected segments during post-processing. However, removal of these segments is not expected to affect insertion accuracy, as clinicians preferentially target long, continuous vessels when performing venipuncture.

Average RMSE’s for the NIR arm segmentation and US vessel segmentation steps were 4.5 % and 4.9 %, respectively. The errors were mostly due to inaccuracies in the initialization of the active contours. While the GVF algorithm is relatively more stable to initialization, the accuracy was nevertheless affected in a small number of cases. Currently we are investigating more robust approaches to initialize the GVF contour, including implementing region growing techniques or utilizing the US speckle statistics or Doppler image information.

The NIR vessel motion tracking step resulted in an average RMSE of 3.3 %. The largest tracking error occurred on Subject 4 due to a rapid arm motion that caused the vessel target to be lost. In such circumstances, the vessel target must be reselected. As an alternative to Lucas-Kanade optical flow tracking, dense optical flow algorithms [10] could potentially be implemented to enforce spatial smoothness constraints on the optical flow fields. It may also be possible to adapt the non-rigid point registration approach used here for stereo registration to perform motion tracking. However, the increased robustness of these alternative techniques will need to be weighed against their added computational costs.

NIR stereo reconstruction errors were not assessed in these studies, since such errors are more readily evaluated in conjunction with robotic cannulation. A focus of future studies will be to evaluate the reliability of the stereo correspondence approach under noise by comparing reconstruction errors in human subjects to previous results by our group on patterned tracking targets.

Fig. 5.
figure 5

Cannulation testing in tissue phantoms. (a) NIR+US imaging system coupled with robotic needle manipulator. (b) Phantom conditions used to assess cannulation performance. (c) First-stick accuracy and completion times of unassisted, image-guided, and robotic cannulation. Error bars indicate deviations for 12 trials.

Full size image

3.2 Comparison of Unassisted Manual, NIR+US Guided Manual, and NIR+US Guided Robotic Cannulations in Phantoms

In addition to using NIR+US imaging to assist in manual cannulations, it is also possible to couple the imaging system with a robotic needle insertion mechanism. The main advantage of robotic guidance is the ability to precisely update the position and orientation of the needle in real-time based on image feedback. Compared to manual cannulation, a robotic system may minimize errors due to misalignments between the needle and vessel, vessel rolling and deformation during insertion, and random arm motions. Previously, we described the mechanical design of a robotic venipuncture device [11, 12] (Fig. 5a) and demonstrated early results in multilayered gelatin-based phantoms simulating the mechanical, optical, and ultrasonic properties of human skin, vessels, and blood [13]. However, the full algorithmic framework had not been implemented, and the performance of the robot was not compared to manual techniques.

Here, we evaluated the effects of vessel diameter, tissue elasticity, and skin tone on the first-stick accuracy and completion time of unassisted manual cannulation, NIR+US guided manual cannulation, and NIR+US guided robotic cannulation. Cannulations were performed on phantoms containing vessels with diameters of 0.8, 1.6, and 2.4 mm, tissue elasticities of 5, 15, and 50 kPa, and skin pigmentation matching that of light, medium, and dark toned patients (Fig. 5b). For each condition, 12 trials were conducted, and all manual trials were carried out by an expert clinician. Cannulation success was defined as the collection of at least 1 mL of blood mimicking fluid perfused through the phantom vessels.

The use of NIR+US guidance was observed to increase first-stick accuracy compared to unassisted cannulation, most notably in difficult conditions, i.e. small vessels, low elasticity, and dark skin tone (Fig. 5c). However, completion time was also seen to increase. Meanwhile, robotic cannulation increased accuracy and decreased completion time compared to manual techniques. These findings suggest a potential for improvement in venipuncture accuracy using image-guidance and for increased efficiency using robotic insertion. Additional measures of success, including the distance of the needle tip from the vessel center, the extent of random needle motion within the tissue, and the average blood volume collected, will be assessed in future studies. The effects of insertion parameters, such needle gauge, insertion speed, and insertion angle, will also be investigated. Finally, collaborative features between the imaging system, the robot, and the operator will be further developed to facilitate clinical translation.

4 Conclusion

In this paper, a device for 3D vessel imaging using NIR and US was introduced. Methods to segment and track the vessels, and to estimate 3D structure, were described and evaluated in human imaging trials. As demonstrated through cannulation experiments in tissue phantoms, the device may be used as an assistive tool or coupled with a robotic system for automated venipuncture. Further clinical development will entail a comprehensive investigation into the robustness of the imaging and image guidance approach, as well as the safety and usability of the robotic platform, in patients with difficult veins.