A systematic review of test accuracy studies evaluating molecular micro-satellite instability testing for the detection of individuals with lynch syndrome

The following bibliographic databases were searched using population terms for Lynch syndrome and intervention terms for MSI or IHC: the Cochrane Database of Systematic Reviews, CENTRAL and HTA [all via the Cochrane library]; MEDLINE, MEDLINE In-Process & Other Non-Indexed Citations, Embase and the Health Management Information Consortium [all via Ovid]; Web of Science [including conference proceedings, via Thomson Reuters]. Search results were limited by date from 2006 to current and to English language publications. The full search strategies are available from the authors.

Four key systematic reviews [4, 7, 9, 11] (and other systematic reviews identified by the bibliographic database searches) were screened in order to source further relevant studies published before 2006 and additional studies published after 2006. These four key systematic reviews [4, 7, 9, 11] were examined prior to the start of this review and were judged to have sufficiently robust searching methods to identify relevant studies published before 2006. Studies which cited the included studies were identified using Scopus (Elsevier). The reference lists of all included studies were screened in order to identify any additional relevant studies.

Titles and abstracts of the studies retrieved from the searches were screened, independently by two reviewers, according to the predefined inclusion criteria specified below. Disagreements between reviewers were resolved by discussion, with arbitration from a third reviewer where necessary. Full texts of included titles/abstracts (from bibliographic database searches, and forward and backward citation chasing), and full texts of studies identified from systematic reviews, were obtained. These were screened in the same way as titles and abstracts.

Studies had to be single-gate (also known as diagnostic cohort studies) or two-gate (also known as diagnostic case–control studies) diagnostic test accuracy studies (or a variation of one of these designs) [13]. They had to recruit individuals with colorectal cancer and investigate the diagnostic test accuracy of molecular MSI testing (with or without BRAF V600E mutation testing and with or without MLH1 methylation testing). MSI must have been compared with the reference standard, which was constitutional MMR mutation testing (including DNA sequencing of MLH1, MSH2 and MSH6 and MLPA or another appropriate technique for detecting large genomic abnormalities as a minimum), by providing sufficient data for at least sensitivity to be estimated. Other outcomes (in addition to sensitivity) were: specificity, likelihood ratios (LR+ and LR-), predictive values (PPV and NPV), concordance (with the reference standard), diagnostic yield, and test failure rates. To be included in the review, studies must have been designed for all participants to receive both the index test and reference standard. However, studies recruiting a representative sample of all patients with CRC (including where an age limit was applied), the reference standard may have been applied to all MSI positive-tumours and to a representative (e.g., random) sample of MSI negative-tumours. Abstracts were included if they reported data from an included study that was published in full.

All included studies were given a study identification label: first-author date. Where needed for clarity, included studies are identified by their study identification label in the results and discussion sections below. Data extraction and quality appraisal were conducted, for all included studies, by one reviewer and checked by another. Discrepancies were resolved by discussion with the involvement of a third reviewer where necessary. Extracted data included details of the study’s design and methods, participants’ characteristics and study results. Risk of bias in individual studies was assessed according to criteria in Phase 3 of the QUADAS-2 tool [14].

The data extracted from the included studies was analysed in STATA 13 (StataCorp LP) using the “diagt” command [15]. For single-gate studies that were not based on high-risk samples (including age-limited population studies), and where data permitted, sensitivity, specificity, LR+, LR-, PPV and NPV, diagnostic yield and concordance with the reference standard (with 95% confidence intervals [CIs]) were calculated. However, for the studies based on high-risk samples, sensitivity (with 95% CI) was calculated (spectrum effects that occur when using a high-risk sample have not been found to lead to significant bias in sensitivity estimates for MSI) [9]. Although not considered an outcome, for illustrative purposes, disease prevalence was also calculated for all included studies, based upon data extracted to 2X2 tables, and representing, therefore, the prevalence in the analysed samples rather than the recruited population. Where extracted data resulted in zero counts in one or more cells, one-sided 95% CIs were calculated.

In primary analyses, MSI-Low was considered as a negative index test result. Unclassified variants (mutations of unknown clinical significance with regards Lynch syndrome), where reported, were considered to be negative reference standard results. The main method of synthesis was narrative.

Of the nine study samples included in this review, three report data from a population-based sample, although only Poynter 2008 recruited an apparently unselected CRC population [21]. The other two studies restricted the population by applying a maximum limit to age of diagnosis: Barnetson 2006 applied an age limit of <55 years and Southey 2005 applied a limit of <45 years [16, 23]. All three of these studies used single-gate designs but varied in size with Barnetson 2006 and Poynter 2008 recruiting 1259 participants and 1061 participants respectively but Southey 2005 recruiting only 131 participants [16, 21, 23]. Disease prevalence in the analysed study samples is provided in Table 1 and was similar in Barnetson 2006 and Poynter 2008 and higher in Southey 2005 [16, 21, 23].

The remaining six studies selected participants with CRC who were at high-risk for Lynch syndrome (Caldes 2004, Hendriks 2003, Mueller 2009, Overbeek 2007, Shia 2005 and the other sample in Poynter 2008) [17‐22]. Five of these studies had a single-gate design [17, 19‐22]. The remaining study (Hendriks 2003) was a variation on a two-gate study design; although participants with positive reference standard results were recruited, no reference standard negatives were included [18]. We referred to this as a reference standard positive study design. The six high-risk studies varied in size; the largest study was Poynter 2008 with 172 participants and the smallest study was Hendriks 2003 with 45 participants [18, 21]. Further details on study and participant characteristics, including disease prevalence in the analysed samples, are given in Table 1.

There was a great deal of between-study variation in both the reference standard and in the MSI testing methods as well as in the reporting of methods. For example, in the studies by Poynter 2008, Mueller 2009, and Overbeek 2007 microdissection techniques were not reported [19‐21]. In addition, none of the population-based studies assessed the same panel of markers, with differences existing in both the number and type of markers, see Table 1. Five studies (Barnetson 2006, Southey 2005, Poynter 2008, Mueller 2009, Hendriks 2003) defined tumours tri-modally (i.e. as MSI-H, MSI-L or MSS) [16, 18, 19, 21, 23], two (Overbeek 2007; Shia 2005) defined tumours bi-modally (MSI positive or negative) [20, 22], and Caldes 2004 used a bimodal categorisation but defined tumours as either MSI-H or MSS [17]. The thresholds used to categorise the MSI status of tumours also varied across studies (Table 1), with some studies using positivity of only 20% of markers as the cut-off between MSI-H and MSI-L (Barnetson 2006) [16], and others requiring 50% (Southey 2005) [23], although different numbers of markers were also used in these two studies.

It should also be mentioned that two of the population-based or age-limited studies (Poynter 2008, Barnetson 2006) and three of the high-risk studies (Caldes 2004, Shia 2005, Hendriks 2003) report on unclassified variants (i.e. mutations where the association with Lynch syndrome is unclear) [16‐18, 21, 22]. Notably, all of the nine studies included in this report predate what is now considered to be the definitive interpretation of MMR gene variants [24]. In this review, therefore, unclassified variants have primarily been considered to be reference standard negative results.

None of the included studies displayed any evidence to suggest that they were at high-risk of bias (Table 2). It should be noted that an absence of evidence to suggest that the included studies were at high risk of bias does not mean that the studies were free from bias. In fact, none of the studies adequately reported whether MSI was interpreted without knowledge of the reference standard results, or whether the thresholds used to denote a positive MSI result were pre-specified, so it was unclear whether the conduct of the MSI test would have introduced bias. Likewise, for all of the included studies, it was not clear whether the flow and timing of the study would have introduced bias. In all studies except Hendriks 2003 it was unclear whether the conduct of the reference standard would have introduced bias [18]. Additionally, only Barnetson 2006 and Southey 2005 reported sufficient information to determine that participant selection was unlikely to have introduced bias [16, 23].

Table 3 gives sensitivity and specificity estimates from primary analyses, where unclassified variants were considered to be index test negative results and MSI-L was considered to be an index test negative result (for studies using a tri-modal distribution of MSI). Only three studies contributed estimates of both sensitivity and specificity (Barnetson 2006, Southey 2005, Poynter 2008) [16, 21, 23]. These suggested, in one study (Poynter 2008), that good sensitivity (100.0%, 95% CI 93.9100.0) could be achieved at the expense of specificity (61.1%, 95% CI 57.0, 65.1) [21]. Conversely a second study (Barnetson 2006) achieved good specificity (92.5%, 95% CI 89.1, 95.2) but at the expense of sensitivity (66.7%, 95% CI 47.2, 82.7) [16]. The third study (Southey 2005) had intermediate values of sensitivity (72.2%, 95% CI 46.5, 90.3) and specificity (87.8%, 95% CI 73.8, 95.9) [23]. Although this pattern would be consistent with a threshold effect, it is difficult to establish this because, in addition to using different numbers of unstable markers to denote a positive MSI result, the panel of markers differed between studies (Table 1).

The range of sensitivities in the single gate, high risk samples (Caldes 2004, Mueller 2009, Overbeek 2007, Shia 2005 and the other sample in Poynter 2008) [17, 19‐22] and the reference standard positive study (Hendriks 2003) [18] all fell within the range of sensitivities identified in the three single-gate population-based sample studies (Barnetson 2006, Poynter 2008, Southey 2005) [16, 21, 23].

In secondary analyses, when MSI-L was considered to be a positive index test result, sensitivity increased (for the six study samples where a tri-modal distribution of MSI was used) and specificity decreased (for the three population-based samples). This is unsurprising; including MSI-L as an index test positive essentially decreases the threshold for a positive test result. Fig. 2 illustrates this effect for the three population based studies.

Further analyses were also conducted where unclassified variants were considered to be positive reference standard results. Only Caldes 2004 and Hendriks 2003 provided sufficient data for these analyses [17, 18]. Both of these studies were based on high-risk populations, so only sensitivity estimates were made, and these were largely unchanged from the primary analyses, most likely because of the low absolute numbers of unclassified variants involved.

Pooling of sensitivity and specificity data was considered but rejected, primarily because of the marked between-study methodological and clinical heterogeneity (e.g. differences in sequencing methods, which genes were tested, in techniques used to test for large genomic deletions and alterations, whether unclassified variants were tested and in the number and nature of the microsatellite makers assessed). Further, as only three studies provided both sensitivity and specificity estimates, this precluded the application of potentially useful test accuracy meta-analytic models such as the hierarchical summary ROC model [25]. Although nine study samples provided estimates of sensitivity, pooling of sensitivity alone is not recommended because of the interdependence with specificity [25‐27].

For the population-based sample in Poynter 2008 and for the other two studies that recruited population-based samples (Barnetson 2006 and Southey 2005) [16, 21, 23], LR+, LR−, PPV, NPV, diagnostic yield and concordance with the reference standard were also calculated. These were primarily estimated based on MSI-L being an index test negative result and these results are given in Table 4. None of the studies reported test failure rates.