Target Product Profiles for medical tests: a systematic review of current methods

Details of the full search strategy can be found in Additional file 1. The following electronic databases were searched: MEDLINE, EMBASE, CAB Abstract Online, CINHAL, Global Health, Scopus and Web of Science. The database search was performed in November 2018 and encompassed a combination of key terms such as ‘TPP’, ‘quality by design’, ‘QTPP’ and ‘test’.

The grey literature and websites were also searched using structured methods proposed by Godin et al. [13]. A customised Google search was conducted to identify relevant websites, and then, each of these websites was hand-searched. For each website, the internal search engine was used supplemented with hand-searching to identify potentially relevant references. Duplicates across searches were removed. This search was also conducted in November 2018. For more details, please see Additional file 1: Table 1.3, Table 1.4 and 1.5.

All searches were conducted by PC and peer reviewed by an information specialist.

TPPs written in English for any type of medical test were included (e.g. imaging, in vitro and in vivo medical tests). There were no restrictions in terms of publication date. All publication formats were included except for newsletters and PowerPoint presentations, as these did not report the methods in sufficient detail to review them.

Endnote was used to manage references. Titles and abstracts of the retrieved references were fully screened by PC based on the inclusion criteria, of which a random 10% sample was independently screened by BS. TPPs that met the inclusion criteria at this stage or those for which it was not possible to determine eligibility based on title and abstract were then screened based on the full text. For those references where full text was not available, we contacted authors. All full texts of the eligible TPPs at this stage were screened independently by PC and BS based on the inclusion criteria. The inter-reviewer agreement rate was calculated with Cohen’s κ statistic. For more details, please see Additional file 3: Table 3.1 and 3.2. Where any disagreements occurred, a consensus-based discussion with the other authors (MM, RW) determined whether the reference was eligible or not.

A data extraction spreadsheet was developed including basic descriptive information relating to the TPP (e.g. publication format, disease of interest, targeted clinical setting, funder, time horizon). Further to this, we extracted data on the methodology used to develop the TPP, including details of the input sources (e.g. expert consultation, review of the literature), the reported decision-making process and the stakeholders involved at each stage of the TPP development. As a common decision-making framework was apparent across the TPPs, we summarised the input sources and stakeholders involved for each phase of this process. Where stakeholders and input sources for the drafting phase were not explicitly reported for, we assessed the sources included in each TPP table and the longer descriptions of each test characteristic.

Each TPP was also assessed in terms of the transparency of reporting the adopted input sources, decision-making process and which stakeholder groups were consulted.

All data extraction was conducted independently by PC and AAS, and in case of disagreement, BS, MM and RW resolved any differences.

In addition to the information above, the test characteristics reported within each TPP were extracted and de-duplicated. Based on an existing evaluation framework for tests (the ACCE framework [14]), two reviewers (BS and MM) independently categorised each of the test characteristics under the following outcomes: (1) test definition, (2) analytical performance, (3) clinical validity, (4) clinical utility, (5) regulatory legitimacy and (6) economic acceptability. The category ‘test definition’ specifies the disorder of interest, target population and purpose of the test, and thus, it overlaps with the concept of ‘unmet clinical need’. Therefore, we renamed the outcome ‘test definition’ as ‘unmet clinical need’ to better represent the type of information TPPs provide.

Analytical performance describes the ability of a test to correctly detect and measure a particular analyte (e.g. precision, trueness, analytical sensitivity and specificity, limits of detection) [15, 16]. Clinical validity is defined as ‘the ability of a device to yield results that are correlated with a particular clinical condition or a physiological or pathological process or state’ [15], whilst clinical utility represents the ability of a test to affect relevant health-related outcomes for patients (e.g. improvement in quality of life, longer lifespan) [17].

Some characteristics did not fall within any of the pre-defined categories. Three additional categories were therefore identified to accommodate these additional characteristics: (7) human factors, (8) environmental impact and (9) infrastructural requirements. Human factors are concerned with the interaction between users and devices [18]. Environmental impact encompasses a change to the environment following an interaction with the product [19]. Infrastructural requirements entail ‘the stock of the basic facilities and equipment needed for realizing a product or providing a service’ [20].

Full details of the literature search results are reported in Fig. 1. Forty-four TPPs were deemed eligible for inclusion in the systematic review [8‐10, 21‐61]. Inter-reviewer agreement was high at title and abstract (κ = 96%) and full-text screening (κ = 98%). For more details, please see Additional file 3: Table 3.1 and 3.2.

The included 44 TPPs consisted of 23 reports, 16 journal articles, 4 published TPP tables (a TPP without any background information or context e.g. [34]) and one conference poster. All TPPs provided guidance on developing medical tests to detect infectious diseases. Fourteen of the 44 TPPs focused on neglected tropical diseases (32%) (e.g. soil-transmitted helminths, Chagas disease, human African trypanosomiasis, schistosomiasis, trachoma, taeniasis cysticercisosis) and on tests for vector-borne infections (32%) (e.g. Zika virus, dengue fever, hepatitis C, malaria, E. coli). Other types of infection included sexually transmitted infections (16%, n = 7), respiratory infections (14%, n = 6) (e.g. lower respiratory tract infection, tuberculosis, pneumonia), Ebola virus [57], meningitis [61] and severe febrile illness [8].

Seven of the 44 TPPs were funded by Bill and Melinda Gates Foundation (16%), and three TPPs received funding from WHO [8, 48, 49]. The healthcare setting of interest was mostly low- and middle-income countries. The majority of TPPs did not disclose funding sources (64%, n = 28).

In some TPPs, a time horizon was chosen to represent the timeframe within which achieving the specifications described in the TPP was considered feasible [22, 23, 60]. In one TPP, this was based on a landscape analysis [22]. In another, expected advancements in technologies and knowledge related to a certain field seemed to justify the time horizon considered for the TPP [27]. Of the 44 TPPs identified, 7 reported the time horizon during which the information included in the TPP will be relevant for manufacturers (16%). Of these, 6 TPPs stated a time horizon of 5 years [22, 23, 27, 28, 51, 60], whilst the remaining considered a time horizon of 10 years [29].

A common decision-making framework, consisting of three distinct phases, was apparent across the included TPPs: scoping, drafting and consensus-building. Figure 2 presents the most commonly adopted activities, input sources and engaged stakeholder groups.

Table 1 provides a summary of the stakeholders contributing to each phase. Some of the included TPPs are not included in Table 1 as they did not report any information related to input sources or stakeholder groups [33‐36, 52, 53, 55, 61]. A summary of the input sources reported to have been used at scoping and drafting phase can be found in Additional file 3: Table 3.3.

We will therefore describe the aim of each phase and breakdown the methodology (activities, input sources and stakeholders) used within the included TPPs where reported. For specific details on each included TPP, see Additional file 3: Table 3.4.

Some of the key activities undertaken during the scoping phase included reviewing published literature (n = 6) or available data (n = 1), and introductory meetings with stakeholders (n = 4).

Some authors reported (n = 4) [22, 26, 37, 50] that they had conducted a ‘landscape analysis’, providing information on the disease area of interest, available diagnostic technologies and related characteristics and limitations. These were usually based on interviews with stakeholders and reviews of the literature. Only Toskin et al. [50] conducted a systematic literature review, reporting the databases searched and key words used.

Consultation with experts (68%, n = 15) and the literature (36%, n = 8) were the most commonly sought sources of information during the scoping phase (see Additional file 3: Table 3.3. for a full breakdown). Only one type of source was considered in 15 TPPs (of which 11 was consulting experts), whilst 7 TPPs considered more than one source.

Denkinger et al. [22] mapped the diagnostic ecosystem of interest and then performed a survey to gauge stakeholders’ preferences. Reipold et al. [48] identified the main characteristic categories (e.g. scope, performance, operational characteristics and pricing) to be included in the TPP.

Five TPPs involved a priority-setting exercise which entailed ranking each identified health need [23, 28, 32, 48, 60].

During the scoping phase, a variety of stakeholders were engaged (Table 1).

The first draft of each TPP was usually prepared by either an established working group comprising experts from different organisations [9, 26, 29, 32, 40, 50, 58] or authors of the published TPP. There were two cases where the TPP was drafted by a completely different organisation [51, 57]. The TPP was often revised several times, and in some cases, it was then shortened to ensure it could be easily communicated to different stakeholders [22, 23, 28, 29, 60].

Of the 44 included TPPs, 33 of them reported which input sources were considered during the drafting phase (75%) (Additional file 3: Table 3.3). Common input sources for populating test characteristics were expert consultations (n = 22) and reviews of the literature (n = 22). Some also referred to mathematical models (n = 9), available data (n = 7), guidelines (n = 6) and ‘field observations’ (n = 5). Only one TPP was informed by pooled data from a systematic review [50].

Twenty-six of the 44 TPPs took into consideration more than one type of source at the drafting phase, as opposed to 7 TPPs which only adopted one (Additional file 3: Table 3.3). Meeting inputs were the most common single source (43%, n = 3).

The stakeholders engaged in the drafting phase are reported in Table 1.

Initial agreement with the TPP was often obtained using a survey of the stakeholders (n = 14). The survey either included general questions regarding stakeholders views on the TPP (n = 4) [22, 25, 27, 51] or adopted a Delphi-like approach to provide an initial consensus on various aspects of the TPP (n = 10). A consensus meeting with stakeholders and experts was typically held (n = 11) and a revised TPP generally agreed upon. In some cases, an additional survey was sent to stakeholders on trade-offs between test attributes [48], or on rating key parameters [51, 53]. For 2 TPPs, the final TPP draft was presented to a broader stakeholder base to validate it.

The number of participants invited to the consensus-building meetings varied (< 20 participants: n = 5; between 20 and 50 participants: n = 7). One meeting included 100 participants [27]. For a few of the TPPs, the authors also took part in the consensus meetings [29, 38, 58, 60].

Less than half of the included TPPs reported information on the activities and stakeholders invited to the consensus-building phase (n = 19). The stakeholders engaged in the consensus-building phase are reported in Table 1.

We also assessed the transparency of the TPPs in terms of reporting their methodology (see Additional file 3: Table 3.5). The decision-making process behind the TPP was not reported in over a quarter of the included TPPs (n = 16). Further to this, many failed to report which information sources were considered to populate the TPP (n = 11). Just under half did not report which stakeholders were involved in the development of the TPP (n = 20). Specifically, the name of the organisations stakeholders were part was only reported in 11 TPPs, whilst 9 TPPs mentioned personal details of each stakeholder (20%) and 4 TPPs explained why certain stakeholders were invited [26, 38, 46, 58]. Sixteen TPPs reported the source of funding (36%).

There were some TPPs where the methodology was very clearly reported [26, 28, 38, 60].

After removing duplicates, 140 different test characteristics were reported across the included TPPs. Some features which did not represent test characteristics have been excluded, such as factors relating specifically to the disease in question rather than the test. For more information, please see Test Characteristics Overview Excel spreadsheet (Availability of data and materials). Figure 3 shows the test characteristics most frequently reported (a full list is available in Additional file 2: Table 2.1).

Figure 4 depicts which characteristic categories were reported in the included TPPs. Details on unmet clinical need, analytical performance and clinical validity appeared to be consistently reported; however, regulatory requirements, environmental footprint and clinical utility were less frequently considered.

The purpose of a TPP is to identify, upfront, the essential characteristics of a test for it to fulfil a pre-specified, unmet clinical need. This should, in turn, increase the likelihood that the test will be adopted into clinical practice and reimbursed [1]. A TPP should also account for contextual aspects that might affect the test’s real-world performance [10], defining infrastructural and technical constraints that impact on the implementation into clinical practice. This review shows that TPPs to date have primarily been developed for global health applications, as the main funding organisations are WHO, UNICEF and Bill and Melinda Gates Foundation. The primary focus on infectious diseases may be explained by the remit of the global organisations who fund TPPs. WHO included HIV/AIDS, neglected tropical diseases, tuberculosis and malaria as priority diseases [62]. Further to this, WHO established the ‘R&D Blueprint’, which aims to promote R&D activities (tests, vaccines, medicines) during epidemics [63]. After having identified pathogen to target first, TPPs are usually commissioned to guide the development process of new healthcare products which will address the high-priority pathogen [63].

However, the development of TPPs should not be limited to one specific disease area or clinical setting; the concept of ‘beginning with an end in mind’ embodied by TPPs could support both international and national health decision-makers. This activity should, in turn, stimulate innovation of new tests driven by clinical needs rather than solely by laboratory discoveries. It would also provide manufacturers with greater clarity around test requirements and confidence in the market for developing innovative tests.

In reviewing current methodology for developing TPPs for medical tests, we have identified three key areas where current TPP methodology could be improved: (1) oversight of clinical utility, (2) a focus on price rather than cost-effectiveness and (3) subjectivity of information sources. Here we discuss each limitation and the implications.

Since this study is a systematic review of publicly available literature, a key study limitation is that we have inevitably missed any confidential or unpublished TPPs developed in-house by test manufacturers. Although the results of our online searches did not identify any companies stating that they have developed TPPs for medical tests, we would not expect to find such information on company websites. Anecdotally, however, we have not encountered any formal TPP development activity (by this we mean definition of desirable test characteristics) within the National Institute for Health Research Leeds In Vitro Diagnostics Co-operative (NIHR Leeds) MIC industry network.

As there are no guidelines on how TPPs for medical tests should be developed, we did not formally assess risk of bias. We did however appraise the transparency with which the methodology underpinning each TPP had been reported. Unfortunately, it was not possible to fully evaluate the TPP developed by PATH [40] as the online appendices were not accessible. Additionally, due to poor methodological transparency, it was difficult to assess with certainty authorship of TPPs and whether authors of TPPs took part themselves in the consensus-building meetings.

Although there is evidence of a common development framework, this review highlights that there is considerable variability in the methods employed to draft TPPs and inconsistencies in which test characteristics are described. A key issue in reviewing the methods implemented was the lack of transparency in methodology reporting.

Guidance on best practice methods for developing TPPs for medical tests would be highly beneficial. Similarly to the US FDA guidance on TPPs for drugs, a guidance document could be developed for TPPs for medical tests summarising the purpose, attributes of TPPs and which test characteristics should be included.

However, to inform the development of such guidance, future research should focus firstly on how to systematically identify unmet clinical needs underpinning a certain disease area. Monaghan et al. [69] developed a valuable checklist for identifying biomarkers based on literature findings and consultations with experts. We believe that this checklist could be pertinent for the scoping phase underlying TPP development; however, this would need further validation in this specific context.

More research is also required to understand how to better incorporate the assessment of desirable clinical utility and cost-effectiveness of innovative tests into TPPs. One possible way forward could be exploring how and if care pathway analysis and early economic modelling could be integrated into the development of TPPs. Care pathway analysis would provide clarity on the mechanisms by which a test could impact on downstream patient outcomes. Early economic modelling could be used to define desirable values for certain test characteristics (e.g. test price, diagnostic sensitivity and specificity) based on cost-effectiveness [67]. Therefore, integrating care pathway analysis and early economic modelling into TPP development might provide more evidence-based information to the test developers.

Outside of the actual methodology for developing TPPs, it would be useful to better understand whether manufacturers develop tests strictly in line with TPPs, or whether there are any factors which make this infeasible or challenging. Additionally, we would be interested in which methods manufacturers usually adopt to develop TPPs to assess if there are any differences with the methodological framework we highlighted here. To this end, interviewing test manufacturers might provide interesting insights on the intrinsic value of TPPs for the industry.

Most importantly to ensure that the development of TPPs becomes widespread practice, it would be valuable to explore how TPPs could be integrated into existing regulatory paths for innovation such as the European Union Regulation for In Vitro Diagnostics (Regulation 2017/746) and the US FDA Drug Development and Approval Process. It might then be possible to align test characteristics featured in TPPs with evidence requirements which are relevant for market approval decisions of new medical tests. This, in return, might increase the applicability of TPPs for the industry.