Automated extraction of Biomarker information from pathology reports

Our data model for representing biomarker knowledge is similar to the CDKRM [19], but it contains a new framework for handling IHC reports. Specifically, the CDKRM covers only microscopic pathologic findings from SP reports as model entities, whereas the model developed in the present study can represent biomarker test results extracted from IHC reports. Additionally, our model has the ability to define which biomarker test described in an IHC report represents evidence for which microscopic finding described in an SP report, which helps avoid incorrect assignment of IHC findings to pathology findings described in the SP report of another patient or of the same patient but for a different indication. For example, when the patient shows symptoms suggesting metastatic cancer, the physician may conduct the same IHC test for several tissue slides obtained from the same patient (Fig. 1a and b, Additional file 1: Figure S1a and S1b). The results of such an IHC test may be reported either in an addendum to the original IHC and SP reports (Fig. 1a), or in an entirely new report (Fig. 1b, Additional file 1: Figure S1a and S1b) [24]. Thus, when designing the data representation model, it is important to take into account that the patient may have one or more SP reports with various microscopic findings for different tissue slides.

To facilitate appropriate association between the findings described in IHC reports and those described in SP reports, we assumed that a tissue slide represents the minimum unit for describing a pathologic finding. This assumption is based on the fact that almost every IHC report states the serial number of the target tissue slide at the top of the results of each analysis performed as the guidelines of College of American Pathologist (CAP) (Additional file 1: Figure S1c) [25, 26]. Furthermore, some descriptions of pathologic findings in the SP report explicitly state which tissue slide provides evidence for that particular finding. Thus, we associated IHC findings with microscopic findings based on tissue slide identifiers (TS_IDs), which represent the serial numbers of tissue slides; furthermore, we introduced the tissue slide paragraph (TS_P) to refer to a set of IHC findings corresponding to a single tissue slide.

Parsing IHC reports using TS_P provides another advantage for cases where the same IHC tests are repeated within a certain time interval because previous tests showed unstable results (Additional file 1: Figure S1a and S1b). In such situations, the same tissue slide (i.e., TS_ID) will show different IHC results, which can be traced easily by collecting all TS_Ps pointing to the same TS_ID, and subsequently checking the time stamps.

However, parsing the SP reports according to this model can be challenging. Most SP reports at SNUH describe microscopic findings organized by the anatomical site of the tested tissue sample, not by TS_ID which is consistent with guidelines of CAP (Additional file 2: Figure S2b and S2c) [24]. Therefore, when the SP report describes the microscopic findings in tissue samples from more than two different anatomical sites or in samples of the same tissue but with distinct pathological characteristics, it is not appropriate to simply map all tissue slides to the anatomical site first mentioned in the SP report. Instead, the system needs to determine the anatomical site of origin for each tissue slide. Fortunately, since one tissue sample is typically resected from each anatomical site, the algorithm steps for determining the anatomical site need to be triggered only when the SP report mentions more than two anatomical sites, thereby minimizing the computational burden of this procedure. A scheme describing our data representation model is given in Fig. 1c.

SNUH reports adopt a semi-structured style for describing IHC findings, further classified as list- or table-based style (Additional file 1: Figure S1a and S1b). We designed the information extraction system in consideration of this semi-structured style, to ensure simplicity and accuracy of parsing. After collecting a set of patterns for each style (i.e., list- or table-based) from training set, we defined a context-free grammar to identify which tokens stand for biomarker names (BNs; e.g., p53, p63, Cyclin-D1) and which tokens represent the corresponding test results (TRs; e.g., positive, negative). The parsing also takes note of the boundary of each TS_P tagged with the appropriate TS_ID. Parsing is achieved using either a list parser or a table parser (Fig. 2), and items parsed from IHC reports are associated with corresponding microscopic findings parsed from SP reports based on TS_ID.

Biomarker term normalization handles terminology variation in BNs and ensures that, for example, ‘Wilms’ tumor 1 protein’, ‘WT-1′, ‘Wilms tumor 1-protein’, and ‘WT1 (4)’ are converted into the single representative form ‘Genes, Wilms Tumor’. To facilitate term normalization, we created a BN dictionary for dictionary look-up and text similarity-based search. The dictionary is based on a list of official product names of biomarkers used at SNUH, comprising 184 original BNs, which we annotated with Preferred Terms (P_BN) of Medical Subject Headings (MeSH) codes. If there was no MeSH entry for a given BN, the P_BN assigned by the pathologists was used as the general term. The list of BNs was then automatically expanded with a set of hand-crafted rules to cover simple variations in notation, such as addition of hyphens or white spaces. This type of dictionary expansion was designed to make text similarity-based searching fast and efficient.

The rules applied to expand the BN dictionary (Table 1) are similar to those described by Bravo et al. [11] for term normalization of gene/biomarker protein names in the biomedical literature, except we remove laboratory-related terms included in parenthesis, so that ‘WT-1 (4)’ and ‘WT-1 (Repeat)’ are normalized to ‘WT-1’. Consequently, the number of entries in the expanded BN dictionary was 3370. This BN dictionary was cross-checked by three pathologists at SNUH.

Term variations in TRs were also considered. For example, ‘positive in mesothelial cells’ and ‘positive in tumor cells’ are normalized to ‘Positive’, whereas and ‘focal weak positive’ and ‘a few positive cells’ are normalized to ‘Focal positive’. Unfortunately, as TR normalization requires background knowledge about the tested biomarkers, it was not feasible to create a TR dictionary by applying the same rule-based approach as that used for creating the BN dictionary. Therefore, three pathologists manually created a TR dictionary based on all TRs from training set to assign the normalized TR terms: “Positive”, “Focal Positive”, “Negative”, or “Error” (i.e., failed IHC test or spelling error that precludes normalization of the detected TRs). This TR dictionary was then used for normalization of TR terms by dictionary look-up.

SP reports are parsed and processed in a similar manner as that employed for handling IHC reports. However, unlike IHC reports, SP reports are rather unstructured in terms of reporting style, and thus, the grammar-based parsing approach used for parsing IHC reports was not adequate for parsing SP reports, as it could not account for all patterns of reporting styles employed in SP reports. Fortunately, in most SP reports archived at SNUH, the first few lines typically mention what organ the tissue sample originated from, whereas the subsequent lines describe the pathologic findings for that tissue [24]. Additionally, when pathologic findings are reported for a specific disease, these findings are grouped under the disease name with an indentation which is consistent with recommendation of CAP for visual separation (Additional file 2: Figure S2b and S2c) [24]. We took this behavioral tendency into account when designing the parser, aiming to reduce the search space for each item and improve parsing accuracy. Specifically, we implemented a tree-based parsing approach of the SP reports, whereby the tree root node is designated in terms of the organ name. In the example illustrated in Fig. 3, the organ name “Breast” appears on the first line of the SP report, so the system creates a tree with the root node containing the first line. Because the next line is numbered and starts with the disease name “1. INFILTRATING DUCT CARCINOMA, multiple, residual (see note)”, this line is assigned as the first sub-node of the root node, at the same level as other numbered lines, such as “2. Fibrocystic change with microcalcification”. Because the lines below the first sub-node have indentations, the system assigns them as sub-nodes of the first sub-node. In this way, the system can generate candidate boundaries for organs, diseases, and specific pathologic findings.

In other words, to create a tree with the root node represented by the organ, the parser must recognize the organ name correctly. To implement such a parser, we first created a dictionary based on terms corresponding to organ names from the Systematized Nomenclature of Medicine-Clinical Terms (SNOMED-CT). The system performs a dictionary look-up operation and calculates a look-up score as the linear sum of the Edit distance, string kernel, Jaro-Winkler distance, and soft term frequency-inverse document frequency statistic. If the look-up score is above a certain threshold, the system registers the term into the organ terminology database, which consists of a normalized set of terms describing organs. We used a training set of SP reports to build the organ terminology database according to this method, which was then verified and curated by two clinicians at SNUH. When parsing the SP reports, the system applies a similar approach to recognize and normalize disease names with UMLS disorder names. If a node corresponding to a disease name is found to have a sub-tree, all its sub-nodes are considered to represent detailed microscopic findings regarding that disease. Finally, for recognition of negation and metastasis terms, we used a set of patterns manually created based on the same training set of SP reports.

Biomarker information extracted from IHC reports and corresponding microscopic findings extracted from SP reports are matched according to TS_P and TS_ID.

We obtained 41,772 IHC reports and 40,519 SP reports archived at SNUH between 2007 and 2012 (Table 2). Subsequently, reports created between 2007 and 2011 were used as the training data set, whereas later reports were used as a validation data set. Additionally, in order to create a gold standard for the evaluation, we randomly selected 400 patients for whom we manually annotated the reports (508 IHC reports and the corresponding 831 SP reports) using the BRAT tool [27]. There were three trained annotators. Each document of patient was annotated by two annotators for normalization of BNs and TRs along with recognition of boundaries of entity mentions. Inter-annotator agreement was measured by F-measure [28]. (Additional file 3: Table S1).

For validation, we selected only those IHC reports for which the corresponding SP reports described information regarding only one organ and one diagnosis, so as to assess solely the issue of data extraction itself. The issue of discriminating findings that refer to multiple organs or diagnoses listed in the SP report will be covered in future work, as this aspect also involves the processing of SP report data without corresponding IHC findings (e.g., findings of the gross examination).

The training data was mainly used to generate patterns while making hand-crafted rules for parsing. Also, we used the training set to manually curate the dictionaries for normalization of BNs and TRs. Validation of information extraction was performed on the gold standard set we manually made.