The Biomarker Toolkit — an evidence-based guideline to predict cancer biomarker success and guide development

There are a huge number of biomarker development studies, but most do not reach clinical practice [1‐3]. In the context of cancer clinical research, a biomarker characterises the presence, progression or response to treatment. As biomarkers are implicated in all steps of patient care, there is an urgent need to reduce the translational gap between the bench side and clinic [3‐5].

The exponential rise in the use of molecular profiling techniques, including metabolomics, genomics and proteomics, has not resulted in a corresponding rise in the number of clinically useful biomarkers. This emphasises that patients are yet to benefit from these discoveries. It is therefore vital to understand: (i) why biomarkers stall and (ii) how research focus needs to shift to promote successful utilisation of biomarkers.

Currently, there is no standard method to assess biomarker research trajectory and guide development; this is reflected by the excessive number of stalled biomarkers. There is a plethora of guidelines used to evaluate the quality/guide research design and reporting; however, in most cases, they provide generalised statements for specific study types. For example, STARD contains a list of essential items for reporting diagnostic accuracy studies [6, 7]. Similar to REMARK, STARD is vague in some instances by asking authors to provide enough detail to ‘allow replication’, without specifying what this entails. Replication is an extremely important characteristic enabling the researcher to achieve reproducible results, and meaningful outcomes; however, the lack of specific detail and explanation regarding this point might increase the risk of assessor bias/misjudgement when applying the checklist.

There is a clear need for detailed guidance to promote biomarker progression across the pipeline and avoid unnecessary and excessive research. Hypothesising that biomarkers unlikely to be clinically translated, lack certain attributes/characteristics in comparison to successfully implemented biomarkers, we aim to develop the Biomarker Toolkit. The Toolkit will consist of a checklist with literature-reported attributes/characteristics associated with a successful biomarker. For the purpose of this paper, a successful biomarker is defined as a biomarker that has been approved by national/international guidelines and is clinically used. On the other hand, a stalled biomarker is defined as a biomarker which is currently not clinically utilised/is not recommended for clinical use by national/international guidelines.

We initially identified the attributes of a successful biomarker through a literature search, semi-structured interviews, and a two-stage online Delphi survey with stakeholders. We subsequently validated the toolkit using published literature on breast and colorectal cancer biomarkers.

Upon validation, utilisation of the Biomarker Toolkit will ultimately define the characteristics of successful cancer biomarkers and will therefore begin to shape how biomarker studies are performed. We envision that the development of this novel tool will assist guidance of research trajectory from any stage of development to promote their clinical utilisation.

The Biomarker Toolkit creation consists of two main components (i) development and (ii) validation. Firstly, we describe the development of the Biomarker Toolkit which is achieved by systematic literature search, semi-structured interviews and a two-round Delphi survey. Then we describe the methodology for validating the Biomarker Toolkit, utilising breast and colorectal biomarker literature.

This novel study utilised an extensive systematic literature search and currently established clinical guidelines to identify attributes defining a successful biomarker, creating the Biomarker Toolkit. Successful validation of the toolkit in both colorectal and breast cancer emphasised its potential as a useful tool for clinicians, academics and industry moving towards the identification of more clinically promising biomarkers.

The 126 attributes included in the Biomarker Toolkit were derived from a combination of systematic literature search results and health study-related guidelines. This greatly improved the clarity of the toolkit, by converging a wide range of attributes in a holistic way.

To allow comparison between breast and colorectal cancer publication scores, the same set of attributes were assessed in both cases. No attributes were excluded to prevent risk of model overfitting. Significantly higher scores in the successful group of both breast and colorectal cancer biomarkers reinforce that the toolkit has the potential to distinguish successful from stalled biomarkers, in two independent cancer types (Figs. 1A and 2A).

As the clinical utility category appeared to be the main driving force behind successful biomarker scores, we aimed to study in detail the source of this difference. As indicated in Fig. 2, successful biomarkers appear to have an evidently higher number of clinical utility publications, published within shorter intervals. Publications appear to show a rise in number approximately 5 years upon biomarker discovery (Fig. 2 and Additional file: Fig. S4–6). This strongly highlights the need to build clinical utility study types, into research plans at this stage, if researchers aim to achieve clinical adoption. This also reinforces the critical need to guide researchers in utilising the Biomarker Toolkit to progress candidate biomarkers along the biomarker pipeline, rather than engaging with excessive biomarker discovery.

Specifically, the accuracy of reporting of assay validation metrics showed the greatest difference between successful and stalled biomarkers in both breast and colorectal cancer biomarker cohorts (Additional file: Tables S9 and S10). Experimental procedure description, method optimisation/validation and biospecimen inclusion/exclusion criteria were more frequently reported in successful biomarker publications in contrast to the stalled biomarkers, where they were more frequently omitted (Additional file: Tables S9 and S10). These attributes contributed to the highest impact towards the overall score. In agreement with the online Delphi survey stage 3, the analytical validity category was ranked as the most important across all biomarker types (Additional file: Table S11). This emphasises the importance of test standardisation and assay development which is often associated with assay commercialisation. Commercialisation of a specific test allows test standardisation and intra-inter-lab repeatability, emphasising the importance of these attributes at an early stage during biomarker development [22].

Currently, the Biomarker Toolkit assesses the totality of the literature for a specific biomarker and provides research guidance towards specified aspects of the field. For example, if the biomarker toolkit is evaluating an early-stage biomarker there will naturally be limited data to assess clinical utility in the early stages. Although a low score will be acquired in this category, it will encourage and guide the researcher to address specific clinical utility aspects and help improve their clinical potential and progression along the pipeline.

At this stage, scores were generated assuming equal weights for each attribute being addressed. This might generate a score with limited reliability as different categories might have a greater level of importance. We utilised the ranks from the online Delphi survey to add weights to different groups of attributes (data available upon request). Adding weights did not appear to cause significant changes in p values of the Cox regression models (Additional file: Tables S12 and S13). Thus, we decided to exclude the use of weightings in the calculations, as (i) they do not improve the Cox regression model in both cancer types while (ii) they add an unnecessary complexity to score calculation. Nevertheless, it is clear that different categories have varying levels of importance to each end user dictated by the remit of the biomarker. Future work on the Biomarker Toolkit aims to enable the user to select weightings for each category of attributes to tailor scoring to meet the goals of the end user. At this stage, we validated the Biomarker Toolkit checklist using analytical validity, clinical validity and clinical utility. Rationale was excluded from study validation as we compared biomarkers of the same cancer type. Future versions of the toolkit aim to include quantitative comparative analysis based on biomarker rationale-related attributes, to assist in the prioritisation of biomarkers with a greater unmet need. Most aspects addressed under the rationale category are subjective and will depend on practice, experience, interest of stakeholders, intended country of use, cancer type, biomarker type in other factors. Incorporating these findings into the Biomarker Toolkit scores will enable the user to evaluate biomarker potential, according to the specific needs of the user [23].

Future work also aims to validate the toolkit using industrial collaborator biomarker databases. By assessing additional biomarkers, we can define cut-off values between successful and stalled biomarkers. This threshold will be used as a reference to compare scores of newly/prospectively assessed biomarkers.

In addition to KRAS and BRAF, microsatellite instability (MSI) has also been demonstrated to be a clinically valuable biomarker [24]. A systematic literature search for KRAS and BRAF yielded a total of 132 clinical papers. Future work aims to address additional biomarkers to enrich the score databank and enable model training utilising natural language processing.

This study generated the first version of the toolkit which requires human input; however, the ultimate vision is to automate the Biomarker Toolkit and establish an accessible application. A specific biomarker will be defined by the user, and all relevant publications will be automatically identified and retrieved from databases. All publications will be automatically screened and scored based on the presence of Biomarker Toolkit attributes. In this way, derived scores can be compared between different biomarkers to assess their clinical potential.

There are various barriers preventing biomarkers from being clinically useful, as previously discussed. Biomarker toolkit scores in individual subcategories reflect gaps in research for a specific biomarker and therefore guide research and development, even at an early stage. This will undoubtedly improve the research quality of reporting and promote the clinical utilisation of biomarkers.

The Biomarker Toolkit is designed to evaluate biomarkers which show early promise, are stalled or slowed down or progressing towards later stages of the pipeline. Future work aims to further develop the Toolkit to allow an individual prediction of the likelihood of a biomarker’s success at each stage of the biomarker’s development based on the scores of each individual category.

This study focused on published literature which is a potential limitation. Specifically, as negative findings are not published, they are therefore impossible to assess. It should be noted that scores are based on literature-reported attributes which do not necessarily reflect biomarker performance. Evidently, a high sensitivity and specificity threshold is of vital importance for a successful biomarker however, there were no significant differences in the performance characteristics assessed between successful and stalled biomarkers, suggesting that these measures do not provide a useful discriminator for biomarker success on their own (Fig. 3). In most cases, only positive results are reported in literature irrespective of a biomarker’s clinical potential. This emphasises the importance of the toolkit, which enables the user to assess a wide range of aspects from biomarker publications, in addition to the commonly addressed performance attributes.

Currently, there is no standardised tool to enable the assessment of biomarkers and target research to those more likely to be clinically translated. There are enormous time and cost implications due to excessive research with very little clinical impact. As supported by Savva et al., a systematic literature search identified 77 gastrointestinal cancer biomarker panels, including 25 with an AUROC >0.9. From the identified biomarkers, all except one were stalled at the discovery phase and none were clinically utilised. This supports the presence of a translational gap, despite high clinical biomarker performance [25].

This original paper focuses on the development and validation of the Biomarker Toolkit. This is a novel tool which encourages a multi-dimensional approach to assess cancer biomarker clinical potential, thus improving the fragmented pathway towards biomarker clinical translation.

The Biomarker Toolkit has the potential to act as a framework that funders, researchers, academics, clinicians and industry can all use in a consistent manner to (i) improve research quality, (ii) standardise biomarker research while (iii) assessing which biomarkers have higher potential and should therefore get funded. It also has the power to rescue stalled biomarkers by identifying where the evidence gaps lie and provide a roadmap for ongoing research. Ultimately, with this toolkit, industry, funding bodies and researchers will finally have a way to navigate biomarkers through the challenging journey of clinical translation. This will reduce the cost and time associated with biomarker development.