Global rigid registration of CT to video in laparoscopic liver surgery

The initial rigid registration of the preoperative 3D image and the intraoperative scene has been explored through methods that rely on fiducials, user interaction and through fully automated methods.

Several approaches propose the use of fiducials, either on the patient skin [12] for needle guidance, or on the organ itself [13] for tracking in laparoscopic partial nephrectomy. Another more robust option, which is applicable in laparoscopic interventions, would be to attach metabolisable fluorescent markers on the organ [14]. Such fiducials can be seen in both modalities. However, these strategies are disruptive to the clinical workflow since they require the acquisition of an additional CT or MRI scan immediately before the intervention.

Surface acquisition of the intraoperative scene has been proposed as an alternative. Several strategies have been developed using laser range scanners [6], optically tracked probes [11], time-of-flight (TOF) data [7] and stereo reconstruction [5]. Once the surface is acquired, the clinician is required to delineate salient anatomical features leading to a point-based initial alignment [6, 11] or to a more complex non-rigid optimisation framework [3]. Another option to obtain the rigid alignment is to manually rotate and translate the 3D preoperative image until it fits the intraoperative data [5]. While some level of user interaction is needed for these approaches, it is generally more intuitive and faster to select salient features than to manipulate the six degrees of freedom associated with a rigid transform.

Hybrid methods have been proposed using cone beam CT and fluoroscopy [15] as bridging modalities between the laparoscopic camera and the preoperative CT, which delivers an additional radiation dose to the patient. Feuerstein et al. [16] propose using intraoperative cone beam CT and optical tracking to register directly to the laparoscopic view without using preoperative information. While their methods achieve promising results, they are based on advanced hardware which might not be available in most clinical settings.

Finally, fully automated techniques have been proposed in [7, 17]. Fusaglia et al. [17] developed an exhaustive search over the principal directions of the intraoperative surface, which is acquired using a laparoscopic laser pointer. While their proposed approach is promising, it still introduces additional tools into the clinical workflow. Dos Santos et al. [7] introduced a novel automatic method to establish surface correspondences between the 3D preoperative mesh and the intraoperative surface acquired with a TOF camera in open liver surgery. Their approach was validated on a phantom of the human liver and on an ex-vivo porcine liver with accuracy better than 1 cm and computation time ranging from one minute to 5.5 h. While their phantom validation under deformation from breathing motion can be sufficient for open surgery, livers in laparoscopic interventions undergo significant general deformation due to pneumoperitoneum. Furthermore, it is unclear how both methods [7, 17] would be translated to laparoscopic interventions since they rely on large surfaces of the liver being visible.

While promising results have been achieved in the literature, we aim to develop an image-guidance system which can handle the challenges of laparoscopic interventions and is easy to integrate with the current clinical protocol without additional hardware or advanced cameras. Furthermore, the system should be usable during surgical interventions, with minimal disruption and fast computation.

We propose a fast, semi-automatic method to obtain a global initial alignment between a 3D liver model extracted from the preoperative CT scan and a surface reconstruction of the intraoperative scene.

An existing formulation of shape matching is extended to incorporate an additional constraint based on the contours of the organ (the ridge line—see Fig. 1), which can be identified on both surfaces with high confidence. Once the delineation of the liver ridge line is given in the two modalities, no further user interaction or initialisation is required for the alignment stage. The proposed method is able to robustly estimate a correspondence set between the two surfaces under deformation, sparse data, partial views and realistic noise levels.

We validate our technique in a simulated environment to show robustness to partial data and deformation. Moreover, we provide quantitative results obtained on a liver phantom and qualitative results on retrospective data from a laparoscopic liver resection to illustrate feasibility in a realistic clinical setting.

In order to minimise E(C) from Eq. 1, the shape alignment problem is formulated as graph matching [19]. Each node consists of a candidate correspondence (i.e. \((m,t)_i\)), and each edge connects two nodes (i.e. \((m,t)_i\) and \((m,t)_j\)). Moreover, if pair \((m,t)_i\) corresponds to \((m,t)_j\), the pairwise constraints imposed will quantify how consistent this association is from a geometrical point of view, thus providing weights for the edges. Figure 2 highlights an example of a correct assignment.

An affinity matrix, W, of the graph is built. The weights associated with each node and edge will result in a strongly connected cluster for data with high consistency. On the other hand, outlier nodes will be either weakly linked or linked in an unstructured way. In cases with a high number of outliers, large deformation or symmetry in the data, some wrong correspondences might be included in the main cluster. As a result, the initial correspondence set C is built by choosing for each target feature point \(\{t_s\} \subset T\), the closest k neighbouring descriptors on the moving surface \(\{m_r\} \subset M\), quickly using kd trees. A spectral analysis method [19] is used to obtain the filtered correspondence set \(C_\mathrm{p}\) from the matrix W. In the next few paragraphs, the proposed formulation for W is detailed, which incorporates the additional term based on the liver contours.

The affinity matrix, W, should have values which are non-negative, symmetric and increasing with higher similarity between the correspondences [19]. So, instead of working with distances as in Eq. 1, the pairwise terms are parametrised as consistency measures:where \(\varepsilon \) is a small number to ensure the denominators are not zero, \(c \in [0,1]\) and it quantifies how similar the geodesic distances are between the pairs \((m_i,m_j)\) and \((t_i,t_j)\). A pair of correspondences is consistent from a geometric point of view if the ratio of the geodesic distances on each shape is close to 1 [20]. However, in the presence of non-isometric deformation of the data, correct correspondences might have consistency values, c, lower than 1. So, the non-rigidity of the data is taken into account by using the following function for c:where the parameter \(\sigma \) sets the amount of non-rigidity allowed for the correspondence set. Furthermore, the function g also helps in separating the outliers by lowering the weights of highly unlikely candidate pairs.

Let \(B^M \subset M\) and \(B^T \subset T\) be the contour points on each surface. The closest contour point to each feature point x on either M or T is computed as \(b_x = \hbox {min} (d_g(x,B))\). The expression used for the proposed contour constraint is:where \(\sigma _b\) allows some deformation between the candidate pairs and their corresponding contours. In practice, \(\sigma _b = \sigma _d\) since they both represent variations in geodesic distances illustrating how restrictive the geometric pairwise constraints will be on the data.

Finally, the affinity matrix, W, is built by placing the unary terms (similarity between descriptors \(\hbox {sim}(f(m_i),f(t_i))\)) on the main diagonal and the pairwise constraints on the off-diagonal:where \(\alpha \) allows different weights for the importance of the two pairwise constraints.

Spectral analysis on the initial formulation for E(C) (Eq. 1) enforces high similarity between nodes \((m,t)_i\) and \((m,t)_j\), as well as approximately equal distances between them (\(d_g(m_i,m_j)\) and \(d_g(t_i,t_j)\)). In addition, the proposed term weighs the edge connecting \((m,t)_i\) and \((m,t)_j\) higher if the distances \(d_g(m_i,b_{mi}), d_g(m_j,b_{mj})\) are similar to \(d_g(t_i,b_{ti}), d_g(t_j,b_{tj})\), respectively. As a result, the estimated correspondence set, \(C_\mathrm{p}\), is explicitly constrained to be consistent with the liver contours on both surfaces, M and T.

Reliable landmarks are difficult to identify consistently between the two surfaces. The strategies used are farthest point sampling [21] and normal space sampling [22]. The former approach was chosen for a uniform distribution of the feature points on the surface. The latter aims to select samples such that the normals are distributed as evenly as possible, thus having fewer points in flat regions. The search space is constrained to only select features on the visible surface of the moving mesh, M, in order to eliminate unfeasible solutions (see Fig. 1). TOLDI was chosen as a feature descriptor, because it was shown to be robust to data sampled irregularly (which is the case for multiple stereo reconstruction surfaces merged together), robust to varying levels of noise and invariant to rigid transformations [23].

The geodesic distance represents the shortest distance on the surface between two points. If the surface changes topology through holes or irregularities in the data, the geodesic distances might become unreliable. Another failure case would be observed if distant parts of the object come into contact and create new shortest paths between feature points. However, it is unlikely the liver shape will change topology in the initial stages of the surgery.

The intraoperative data collected during laparoscopic surgery will most likely have some degree of sparsity—sparse point clouds [3], sparse data collection [4], sparse stereo reconstructed patches [5].

So, let S be a smooth interpolated surface of the intraoperative point cloud, T. The target feature points, \(\{t_s\}\), and contour points, \(B^T\), can be expressed on the interpolated surface with nearest neighbour computation. The geodesic distances on S are computed using the fast marching algorithm [24, 25]. This step is the most computationally expensive to compute in our implementation. However, faster alternatives can be employed [26].

The proposed shape matching technique starts with a large set of correspondences, C, and spectral analysis prunes out the outliers, resulting in \(C_\mathrm{p}\). The final set of correspondences is not guaranteed to consist only of correct matches, especially in cases with significant deformation.

Random sampling and consensus (RANSAC) [27] (see Fig. 1) is used to get the best minimal solution \(\{(m,t)_i\} \in C_\mathrm{p}\) out of the pruned set of correspondences \(C_\mathrm{p}\). The final pairs \(\{(m,t)_i\}\) are used for the least squares estimation of rotation and translation. The estimated transformation is considered to be a good fit if the root-mean-squared error (RMSE) between the target and moving models is less than a threshold \(d_\mathrm{RANSAC}\) and the difference between the normals is less than a specific angle threshold \(a_{\mathrm{normals}}\): dot\((n_m,n_t) < \hbox {cos}(a_\mathrm{normals})\quad \forall \; {(n_m,n_t)_i} \in C_\mathrm{p}\).

We validate the robustness of the proposed method to partial views of the liver and to increasing deformation levels. The mesh of a liver phantom (OpenCAS [8]) is used as the moving model, M. The mean distance between the estimated registration result and ground truth vertex correspondences is measured. Three algorithms are compared:

The proposed method is validated using the OpenCAS [8] public dataset, which contains 3D meshes from an experiment in which a silicone liver phantom is deformed by an indentation. The positions of small Teflon marker balls in both the initial and deformed states of the phantom are given. Please refer to Suwelack et al. [8] for more details about how the dataset was built.

The 3D model of the liver phantom in its initial state is used as the moving model, M. The proposed coarse registration method is tested in two scenarios. Firstly, a partial view of the deformed liver phantom is used as the target model, T (Fig. 4-left), which tests the performance under deformation, partial and sparse data. Secondly, a partial surface reconstructed from an intraoperative stereo endoscopic camera (Fig. 4-right) is used as T. On top of deformation and partial data, this scenario also tests the proposed method in realistic noise levels from a stereo reconstruction. After the global alignment is estimated with the proposed method, Levenberg–Marquardt iterative closest point (LM-ICP)[10] is applied. The distribution of target registration error (TRE) in mm is computed for both cases. The mean TRE for the partial deformed surface is 7.94 mm after our proposed global alignment, which is further reduced to 7.77 mm after LM-ICP. Similarly, in the case of the intraoperative partial surface, the mean TRE of 28.62 mm after the global alignment is decreased to 12.10 mm after LM-ICP. The best case scenario for rigid registration would be point-based alignment of the marker ball positions before and after deformation, with a mean TRE of 5.66 mm.

The proposed approach is demonstrated on clinical data from a video sequence acquired during a laparoscopic liver resection. The 3D mesh of the liver surface was extracted from a CT scan before surgery.² We use the SmartLiver system [5] to process the retrospective data. The liver is automatically segmented in the laparoscopic video with a deep learning framework [30]. Surface patches are collected to cover all the visible surface in each video [31]. They are consequently merged together using optical tracking data.

Figure 5 shows the visual assessment between the manual alignment performed on the SmartLiver GUI and the proposed method. The last column illustrates an example of augmented reality in laparoscopic liver surgery after LM-ICP is applied to the proposed alignment.