Setting Standards for Reporting and Quantification in Fluorescence-Guided Surgery

The CalibrationDisk™ (SurgVision,‘t Harde, the Netherlands) is a calibration device for FI systems. The disk can hold eight clear polypropylene tubes of 0.65 ml (Catalog # 15160, Sorenson, BioScience, Inc., Murray, USA) (Fig. 1). The device consists of two parts: an upper disk which holds the tubes in place and a base on which the upper disk can rotate. The upper disk has round windows that allow measurement of signal intensity in each tube. By rotating the disk, different concentrations of a tracer can be imaged at the same position and under the same excitation conditions providing assessment of homogeneity in illumination of the field of view. We performed the experiment with two different commercially available imaging systems with distinctly different modes of operation: imaging system A and imaging system B. Both systems are state-of-the-art clinical systems optimized for intraoperative NIR imaging providing real-time fluorescence images and white light overlays. System A, a cooled system, has two cameras, one for white-light image acquisition and one for fluorescence image acquisition of a single NIR channel (825 to 850 nm). System B uses a single camera for imaging two fluorescence channels (far red 700 to 830 nm and NIR 830 to 1100 nm) and a white-light channel. Four different NIR fluorescent agents including two dyes (indocyanine green (ICG) [Pulsion Medical Systems Munich, Germany] and IRDye 800CW[LI-COR Biosciences, Lincoln, NE, USA]) and two molecularly targeting fluorescent tracers (bevacuzimab-IRDye800CW [21] and a folate-NIR fluorophore (OTL-38) [3]) were used to make dilution series in Intralipid 2 %. All dilution series consisted of 21 concentrations, starting at 10,000 nM and, following one on one dilution with Intralipid 2 %, ending at 10 pM. Vials containing the 21 different concentrations were divided into three sets of seven (low, medium, and high concentration). A background or “0-vial” containing Intralipid 2 % without a fluorescent agent was added to each set (Fig. 1).

As a reference, the performance of the two intraoperative FI systems was compared to the Pearl Impulse preclinical imager (LI-COR Biosciences, Lincoln, NE, USA). This system contains an ambient-light-free chamber and can be used as a standard of the maximal linearity and sensitivity achievable. Analysis of images obtained with the Pearl imaging system was done using the software suite provided with the device.

We evaluated 271 images available from previous studies to evaluate the effect of ROI selection and background noise on SBR. We randomly selected a representative sample of intraoperative and ex vivo images from both animal and human studies in different tumor types using different fluorescent agents and imaging systems (Table 2).

On these images, we drew a ROI around the (histologically confirmed) tumor. To evaluate the effect of ROI selection, we drew two different ROIs of similar area size in the background: the darkest region adjacent to the tumor ROI and the lightest region adjacent to the tumor. Lastly, we drew a ROI using our preferred method selecting the region surrounding the tumor ROI remaining within the anatomical structure in which the tumor is present. This can be done in ImageJ by subtracting the tumor ROI from the overlapping background, using the ROI manager menu and selecting the “More” button followed by the “XOR” button. Figure 2 displays a representative example of ROI selection. Mean gray values and the standard deviation of the pixels within one ROI were assessed using ImageJ. To evaluate the effect of background noise, we applied both SBR and CNR equation on the values obtained with ImageJ.

By assessing the lowest concentration visible with an imaging system, sensitivity of the system can be determined. Using the CalibrationDisk™, the lowest detectable concentration can easily be assessed for each agent and imaging system (Fig. 3). We found, irrespective of the fluorescent agent used, that system A is superior to system B in terms of sensitivity. The lowest detectable concentration with system A is 1 nM, for system B this is 500 nM. For comparison, the Pearl Impulse detects concentrations as low as 0.05 nM. Gain and exposure time settings influenced sensitivity of the system, with high settings leading to maximal sensitivity for both systems, nevertheless these settings did not affect the mutual differences between systems A and B.

Analysis of the linearity of imaging systems A and B reveals striking differences, with system A being superior, approaching the linearity of the Pearl Impulse. For the low and medium concentrations, the detection limit of the system was reached; therefore, signals measured by system B remain in the same range resulting in a horizontal line on the log10 graph. However, for the high concentrations, system B does display a linear gradient similar to system A and the Pearl Impulse (Fig. 4).

The method applied for ROI selection had a profound influence on both SBR and CNR. Figure 2 shows the influence of background selection on CNR and TBR. Obviously, selection of a darker background will increase the SBR. As Fig. 2 effectively displays, a sufficient SBR (> 2) can be achieved by adapting background ROI selection. In addition, the area size of the background ROI influences the CNR. As the selection of a small area as background results in a small standard deviation, CNR is higher when smaller ROIs are selected.

Sensitivity of two different imaging systems for intraoperative use was assessed. The goal of our experiment was not to quantify performance of an individual imaging systems, but to demonstrate how to compare different imaging systems and predict clinical performance in an experimental setting. Hence, we decided to select systems with a distinct mechanism of action and anonymize both systems. For the imaging system assessment, we used the CalibrationDisk™ and tubes filled with descending concentrations of different fluorescent agents. Various types of other phantoms are described in literature. The use of solid polyurethane phantoms, with TiO2 particles mimicking scattering and quantum dots mimicking different concentrations of a fluorophore, is suggested by Zhu et al. [9]. Benefit of these solid phantoms is their longer shelf life that allows repeated measurements over time. Disadvantages are that these phantoms are difficult to construct. More importantly, however, is that while quantum dots mimic the fluorescence of the agent, it does not use the actual fluorescent tracer that will be used in humans and thus provides only a distal proxy of the crucial information. Others have suggested the use of more tissue-like phantoms made from gelatin [22, 23]. The fluorescent inclusions, used to mimic tumors, are prepared using a custom-made silicone mold, which is filled with agarose mixture containing a relevant concentration of the fluorescent agent. Alternatively, hydroxyapatite (HA) crystals loaded with Pam78, a fluorescent derivative of the bisphosphonate pamidronate, calibrated against the relevant concentration of the fluorescent agent can be used. Background tissue is made from gelatin to which various ingredients can be added to mimic absorption (hemoglobin or pink India ink), scattering (using Intralipid or milk powder), and autofluorescence (ICG). The fluorescent inclusions can be incorporated at various depths in the gelatin base. Although close to the clinical setting, the manufacturing of these tissue-like phantoms is laborious and the shelf-life is limited. For training purposes, these phantoms are probably superior, but for sensitivity and comparability testing of imaging systems most features seem superfluous. The use of the CalibrationDisk™ allowed determination of the lowest concentration detectable and the ability to quantify concentrations. This data can be used to compare imaging systems or to predict clinical performance and can consequently simplify the task of selecting the right system for a certain application. However, users of imaging systems should be aware that the generated in vitro data is a simplification of the in vivo reality. In vivo optical tissue properties influence the ability to discriminate the signal from its background, sufficient knowledge and careful consideration of these limitations remains of the utmost importance. Moreover, various other factors besides sensitivity may play a role in the selection process. Dsouza et al. suggest six key features for imaging systems [10]: (1) real-time overlay of white-light reflectance and fluorescence images; (2) fluorescence-mode operation with ambient room lighting present; (3) high sensitivity to tracer of interest; (4) ability to quantify fluorophores in situ; (5) ability to image multiple fluorophores simultaneously; and (6) maximized ergonomic use.

Although we focused on the sensitivity point, the other points are equally relevant when deciding on the optimal imaging system for a certain application. In addition, the costs should also be taken into account.

The basic principle for FI is the excitation of fluorophores using a light source and the subsequent detection of photons emitted by the excited fluorophore using the imaging system. The detection of emitted photons is influenced by tissue optical properties like absorption and scattering (including reflection). Absorption of photons is a consequence of tissue specific absorption properties, of which in humans blood is the main absorber [24]. Scattering is the change of the direction of a photon in tissue. Scattering events can cause decreased signal strength and source localization, as occurs with fatty tissue. The effect of these phenomena is increased when a photon has to travel through more tissue, thus with greater tissue depths. Reflection seen at the surface of tissues causes diffusion of the signal and consequently reduced detection. Besides the targeted fluorophore, excitation can cause endogenous fluorophores within the tissue to fluoresce as well (e.g., autofluorescence). Noise is the sum of autofluorescence, scattering and reflection events and can make it difficult to discern the actual fluorescence signal. A SBR > 2 is generally considered adequate to differentiate target from background [25]. Nevertheless, there are clinical trials in which SBR values below 2 are described as sufficient for intraoperative FI [26]. Despite routine use in optical imaging, including nuclear medicine, the cut-of value of 2 seems to be based on marginal evidence and the clinical relevance of this cut-of seems at least questionable. CNR is the ratio of the absolute difference between background and tumor signal and the standard deviation of the background. Rewriting the CNR formula shows that the CNR is strongly dependent on the SBR:

The use of CNR is theoretically favorable over SBR to quantitate in vivo obtained (patient) imaging data as it is more comprehensive measure and thus provides extra information. Following the three-sigma-rule, an empirical statistic rule often used in descriptive statistics, a CNR of 3 or higher indicates that the average tumor signal is present only in approximately 0.135 % of the background selection [27]. It could therefore be argued that the CNR has a more evidence-based cut-off value.

However, this does not instantaneously mean that the CNR can be declared superior to SBR, as both quantitative measures are critically dependent on the ROI selection. Background ROI selection has an important influence on the mean background signal (SBR and CNR) and its standard deviation (CNR), rendering CNR is equally prone to selection bias as SBR. To increase reproducibility in FI research ROI selection process should be standardized and described more in detail in scientific articles. We suggest a ROI selection procedure that is representative and least prone to selection bias. However, as selection is done manually, bias cannot be excluded completely. The gold standard remains performance of biodistribution studies describing the percentage of the dose per gram of tumor and background tissue.