Primary histologic diagnosis using automated whole slide imaging: a validation study

Automated, high-speed, high-resolution whole side imaging (WSI) robots, introduced as prototypes only a few years ago, are becoming increasingly robust and capable. Current devices can automatically image several hundred slides per day and these volumes will only increase in the future. With time, this technology should have a significant and beneficial impact on pathology practice as it will enable pathologists to better apply computational power and network connectivity to all aspects of their practice, not just reporting and accessioning. Though several years off, integrated digitization (gross, slides and text) could streamline workflows, increase productivity, and allow new and unforeseen capabilities such as computer-aided diagnosis (CAD). Despite these promises, however, there exists only a relatively small number of papers describing automated WSI in education and research [1‐6], far fewer directly comparing automated WSI with traditional microscope examination in diagnosis [7‐11], and virtually none describing automated WSI in clinical practice. A good review of WSI technology and applications can be found in Gu and Ogilvie [12].

Clinical (diagnostic) validation of WSI systems is extremely important because image capture is automated, focusing mechanisms are novel, and the size of the datasets they generate mandates the use of lossy compression. Validation studies are also an important way for pathologists to guide the development of the technology. There is little question that digitization itself is not a barrier to pathology diagnosis; for many years pathology practices have provided remote diagnostic services using robotic real-time, near real-time and even static image-based telepathology. For example, Dr. Bruce Dunn and his colleagues at the Milwaukee Veterans Administration reported thousands of successful diagnostic robotic telepathology sessions with a remote hospital in Iron Mountain, MI [13]; Dr. Keith Kaplan established an active robotic telepathology service that now supports 22 army facilities [14]; and Drs. Marida Minervini and Jake Demetrius and their colleagues in Pittsburgh used a static telepathology system to support transplant pathology at a hospital in Palermo Italy [15]. In these traditional telepathology applications, however, pathologists directly control the image formation (focus) and capture process. This is not the case with high-speed WSI systems in which image formation and capture are automated.

Given the large number of slides generated by pathology laboratories, one of the most essential features of a clinical WSI system is slide capture speed (or throughput). Fortunately, modern WSI devices are becoming very fast, with current devices ranging from 2 to 8 minutes per slide. To achieve this speed, systems employ a unique focusing strategy. While some WSI systems can be set to auto-focus at every point on the slide [16‐18], this is a time consuming process. Systems designed for faster speeds tend to pre-focus on a limited number of points and use proprietary algorithms to calculate an "in focus surface" above the slide such that, if the objective lens follows that surface during capture, the resulting image should be in focus. While engineers are developing increasingly sophisticated and successful algorithms, and increasingly capable stages for implementing these algorithms, perfect focus is rarely achieved over an entire slide. This is largely because tissue sections are not planar, especially compared to the depth of field produced by high numerical aperture 20 × or 40 × objectives. Some researchers have suggested that vendors decrease the numerical aperture of their lenses or stop down the condenser – effectively trading lower maximal optical resolution for a deeper depth of field [19]. However, achieving perfect focus across an entire slide remains a major challenge for modern high-speed WSI systems and a potential limitation to their clinical use. Therefore, the high degree of diagnostic capability expected from a traditional, microscope-based pathology examination and reported for robotic telepathology systems (vide infra) [13‐15] cannot be assumed in automated WSI and must be validated in controlled studies.

In a WSI system, the optics form an image, the focusing system places the image precisely on the sensor (usually a CCD) and the CCD/camera samples that image. Captured images are generally subjected to compression and are viewed on a wide range of monitors. While the quality of focus is often considered the most critical aspect of image quality, these other components (and their interactions) are also important. In actual practice, if WSI systems have quality optics with sufficient numerical aperture (as found in most current devices), and if precise focus is achieved (vide supra), then the quality of the captured image will be determined by the quality of the image sampling. Sampling is limited by the spatial sampling period ("pixel resolution") [20] and dynamic range of the sensor. Spatial sampling period is calculated by dividing the size of a pixel by the primary optical magnification of the system. For example, a CCD with 6.6 micron pixels and 20 × objective lens gives a sampling period of 333 nanometers. While manufacturers (and users) often tout spatial sampling period as a proxy for image quality, in reality captured image quality is a function of many parameters (including optical resolution, focus, spatial sampling period and dynamic range) and relationships between these variables are complex issues well beyond the scope of this paper [21]. To improve image quality, for example, one may choose to decrease (improve) the spatial sampling period by increasing the magnification or decreasing the pixel size. However, these changes can have unintended consequences: increasing magnification often increases numeric aperture which could affect focus while smaller pixels could decrease dynamic range and actually decrease image quality. The point is that relationships between image quality, optical resolution, sampling period, dynamic range, etc. are complex and ideal combinations cannot be easily predicted. Again, careful study, testing and validation are needed to guide development and practice.

Another important parameter that impacts image quality is the use of compression. The effect of compression on traditional static and dynamic telepathology has been well studied [22]. A typical WSI system with a sampling period of 0.33 um/pixel generates 900 million image pixels per square centimeter of tissue section. At 24 bit color, this is 2.7 GB of data per square centimeter. Therefore, all high-speed WSI devices use lossy compression as a matter of course. Although it is our experience that "reasonable" compression, especially using JPG or JG2000 algorithms, provides excellent image quality with limited artifact, the effect of compression has not been carefully studied on WSI diagnosis and its clinical impact (or lack there of) should not be taken on faith.

These considerations and many others make it clear that, for automated, high-speed WSI to reach its potential in pathology practice, it must be continuously evaluated and validated by the pathology community. In this paper we present a retrospective validation study based on 25 randomly selected genitourinary and dermatopathology cases. Our goal was to determine if pathologists could effectively sign-out complex cases (full final diagnosis field with comment) exclusively through whole slide images captured by a modern imaging robot. We also wanted to establish a level of baseline diagnostic capability for pathologists evaluating WSI for their practices and to identify problem areas that could be addressed by future focused studies.

The main goal of this study was to evaluate the clinical utility of current automated, high-speed, high-resolution, WSI robots by evaluating the ability of pathologists to sign-out complex cases exclusively through digital slides created by those devices. The study had two related components. In the first, three study pathologists independently "signed-out" 25 genitourinary pathology and dermatopathology cases exclusively with digital slides. Each pathologist composed a complete final diagnosis including diagnostic comments when appropriate. The work was done through our institution's LIS workflow and the pathologists could order special stains and recuts. The reports from the different pathologists were then compared, discrepancies examined, and a consensus diagnosis was rendered. In the second part of the study, the WSI consensus diagnosis was compared with the microscope-based, sign-out report of the original case pathologist.

The comparison of the WSI consensus diagnosis with the original microscope-based, sign-out report had a remarkable and unexpected result. There was virtually complete diagnostic agreement between the WSI-based report and the microscope-based report for all 31 surgical parts of the 25 cases examined. The results seem to indicate that the image information contained in current whole slide images is sufficient for pathologists to make reliable diagnostic decisions and complex diagnostic reports. This should be very encouraging to the WSI industry; however, these results should not be interpreted as implying that automated, high-speed whole slide images are "as good as" the microscope. In fact, every whole slide image in the study had a least some areas of imperfect focus, and some had what appeared to be compression artifact. In other words, what seems to be impressive diagnostic capability reported in this study should not be construed to mean the images obtained in the study were perfect, or that they were as good as the images formed on the retinal by a well-aligned microscope.

The comparison between the individual WSI-based reports of the study pathologists was also encouraging. In 17 of the 25 cases (21 of 31 specimens/parts) the reports of the study pathologists showed complete agreement. Of the 12 documented discrepancies (in 10 specimens/parts), 9 did not appear to involve image quality issues but three did (vide supra, Results). Looking at this in another way, there were three pathologists who independently rendered extensive diagnoses on 31 parts (93 diagnostic events). Of these 93 events, image quality and/or interpretation issues were implicated, at least partially, in 3 (in cases 38, 50 & 57). It is these cases that are the most interesting and informative. The majority of these discrepancies involved largely cytologic decisions along the continuum of benign, atypical, suspicious to malignant, and while the WSI consensus arrived at the correct diagnosis, the pathologists agreed that examination under the microscope made the decision much easier.

Examining these discrepancies carefully, one can identify some degree of poor focus in the areas in question resulting in blurring of nuclear details. In some cases this may be exacerbated by compression artifact and some degree of limited dynamic range (causing dark area, like nuclei, to appear darker than under the microscope). It is very likely that these effects interact with each other (i.e. poor focus blurs features that may be further eroded by compression). While compression artifact can be improved by changing compression parameters, methods for focus and dynamic range appear to require further development for automated, high-speed WSI systems to be used confidently in clinical primary diagnosis. We are examining these features carefully with controlled experiments and will report our findings in a separate publication.

It is important to note that these effects, though they were identified as causes of discrepancies in a relatively small number of cases and did not cause a single mis-diagnosis in consensus, could, in fact, be seen in the great majority of slide images in this study. Virtually every whole slide image had some areas of limited focus and some had areas of what appeared to be compression artifact. However, pathologists could "read around" these areas to find areas of excellent image quality, and they quickly learned how to interpret these "artifacts". For example, Figure 27 shows an area in a prostate biopsy with poor focus and other artifacts (it is from a whole slide image that had excellent focus over large areas). Given the ability to evaluate the entire slide, the pathologists had no problem identifying this area, correctly, as cancer.

The study was designed to measure a very specific thing: the ability of pathology to sign-out completed cases using currently available commercial WSI technology. This study was conducted using an Aperio "T2" high-speed, high-resolution, automated WSI robot. We have several different WSI devices in our lab, from a number of vendors. We chose the Aperio for this study because Aperio is the market leader today, making it a good example of a "typical" modern, WSI device. It is our experience that the results of this study, the good and the bad, are to some degree representative of all automated, high-speed devices we have seen, and they spring from the same limitations that all high-speed devices share – the challenges of focus, speed, and compression.

The study involved a small number of pathologists and a relatively small number of cases. Its interpretation required the comparison of complex, free text diagnostic reports. These limitations make it difficult to generate meaningful statistics and should be taken into account when evaluating the conclusions presented. It should also be noted that the design of a study can significantly impact its results. In the current evaluation, we did a line-by-line comparison of the written reports of study pathologists using WSI, then had the same pathologists develop a written consensus WSI report. We then compared that report, line by line, with the original microscope-based, sign-out clinical report. We felt that this approach allowed us to examine the variation between study pathologists, and, at the same time, to compare a consensus opinion against a clinical "gold standard". Other approaches were possible. For example, we could have randomly assigned pathologists to a modality (WSI or microscope), compared reports, and changed assignments for every case. Given the range of experience in our pathologists (and the limited number of case), this could have caused significant bias and would not have given us a gold standard report to judge against. Alternatively, we could have had the same pathologists see the same cases with both modalities, separated by a time period long enough for the pathologists to forget their initial impression. However, given the complexity of some of the cases and the need for diagnostic comments, recuts and special studies on a significant number of the cases, we felt that the required latency time period would be prohibitively long, especially given that rapid technical progress of whole slide imaging systems. It is also important to note that, in comparison studies, the way that "agreement" and "disagreement" are judged (for example, line by line comparison of written reports or a simple "benign versus malignant"), clearly affects the reported results. Because of this real effect, we decided to make all of our data available in a supplemental file (Appendix 1) so that readers can better interpret the findings for themselves. In the future, to better assess the capabilities of current WSI technology and to validate our findings we plan to build on this study, including more controlled assessments of conditions that may impact the performance, user attributes and application of WSI across a variety of clinical environments.

Automated, high-speed, high-resolution WSI is an evolving technology that holds great promise for pathology practice. At least in this study, the results indicated that images produced by current devices have enough image information to allow pathologists to produce accurate, complex, and detailed diagnostic reports, even for difficult and complex cases. It is important, however, to recognize that current images have significant limitations that can cause diagnostic confusion, including areas of sub-optimal focus and artifacts that appear to be related to over-compression and limited dynamic range. These limitations must be kept in mind if WSI is to be used for routine clinical primary diagnosis or other clinical applications such as quality assurance or second opinion collaboration. We are beginning to understand these technologies more clearly and this should allow pathologists to better use, and manufactures to better build, WSI systems in the future.

The University of Pittsburgh Investigational Review Board reviewed this research study, Number 0501128, and approved it as exempt from Federal Policy for the Protection of Human Research Subjects.