Background
Methods
Study design
Participants
-
Working in a research department or unit clearly identified as a conducting translational oncological research (e.g. translational research platform of a cancer hospital)
-
Involved in a consortium clearly identified as a translational cancer research consortium (e.g. EurocanPlatform or ERA-NET on translational cancer research)
-
Having presented their work in a translational cancer research conference (e.g. the ‘translational research’ session of the European Society for Medical Oncology or American Society of Clinical Oncology conference)
-
Teaching in a course or training on translational research in oncology
Interviews
Survey
Ethics statement
Results
Characteristics of participants
Participants in the qualitative study
Qualitative study n (%) (n = 23)
|
Delphi first round n (%) (n = 35)
| |
---|---|---|
Years of experience in cancer research | ||
5–9 | 7 (30) | 5 (14) |
10–14 | 2 (9) | 4 (12) |
15–19 | 5 (22) | 4 (12) |
20–24 | 2 (9) | 11 (31) |
25–29 | 5 (21) | 6 (17) |
≥30 | 2 (9) | 5 (14) |
Sex | ||
Male | 16 (70) | 24 (69) |
Female | 7 (30) | 11 (31) |
Institution of affiliation type | ||
Comprehensive cancer centre | 11 (48) | 17 (49) |
University hospital | 6 (26) | 11 (31) |
Public research institute (not affiliated to a hospital or cancer centre) | 2 (9) | 4 (11) |
National agency | 2 (9) | 0 |
International or European agency | 2 (9) | 0 |
Industry (pharmaceutical or consulting) | 0 | 3 (9) |
Country | ||
French institution | 18 (79) | 3 (9) |
Italian institution | 1 (4) | 8 (23) |
Dutch institution | 0 | 6 (17) |
British institution | 0 | 4 (11) |
Belgian institution | 0 | 3 (9) |
Other European country institution | 3 (13) | 11 (31) |
International institution | 1 (4) | 0 |
Training | ||
Masters degree (MSc) | 0 | 1 (3) |
Medical degree (MD) | 2 (9) | 7 (20) |
Medical degree and MSc | 1 (4) | 0 |
Medical degree and PhD | 7 (30) | 14 (40) |
Pharmacy degree (PharmaD) | 1 (4) | 0 |
Pharmacy degree and PhD | 5 (22) | 1 (3) |
PhD | 7 (30) | 12 (34) |
PhD or MSc specialisation | 20 | 28 |
Biology | 10 (50) | 13 (46) |
Chemistry or biochemistry | 2 (10) | 2 (7) |
Pharmacology | 2 (10) | 0 |
Physics | 2 (10) | 2 (7) |
Computer sciences | 1 (5) | 0 |
Management of healthcare organisations | 0 | 2 (7) |
Immunology | 1 (5) | 2 (7) |
Genetics | 1 (5) | 3 (10) |
Epidemiology/statistics | 0 | 2 (7) |
Biotechnology | 1 (5) | 1 (4) |
Pathology | 0 | 1 (4) |
Participants in the modified Delphi survey
Themes and subthemes extracted from the qualitative study
Issues to consider when developing an evaluation system (qualitative study)
Definition of translational research: a common basis but unclear limits and scope
“I tend to consider translational research in one way only. From the lab to the clinic. It is true that there might be some disciplines where the other way, from the clinic to the lab, can also be done. But as I tend to consider it one way only, I think we shouldn’t keep on funding laboratories of so-called translational research that don’t translate anything.” (Clinician-researcher, university hospital, 21 years of experience in cancer research)
“People put whatever they want [in the definition of translational research] because it does not really mean anything. It is a trendy word because every other year we need a new fancy word. It used to be ‘transfer research’ then ‘applied research’.” (Clinician-researcher, university hospital, 37 years of experience in cancer research)
“There is much more investment into clinical research than in epidemiological and prevention research. There are two reasons for that: the first is that the public wants a cure to cancer. […] Charities funding cancer research adapt their priorities to the requests of donors. The second reason is financial. Epidemiological research does not profit companies, although it saves the government money. There should be a stronger public support for epidemiological research, as clinical research is already supported by industries. Charities and the public should be more biased towards epidemiological research.” (Researcher, public research institute, 26 years of experience in cancer research)
Multidisciplinarity and collaboration are crucial for the successful conduct of translational research
“Sometimes people who do fundamental research, it is good, they have ideas, but eventually, they have no idea about what is a patient, or the same difficulties [we encounter as clinicians]. And I think that the clinician’s perspective can make research more practical, more applied, closer to the issues. On the contrary, those who carry out fundamental research, because they don’t have this commitment towards the patient, are maybe more reasonable regarding what is possible and not possible, more practical in the delays of application […]. Or on the contrary more unreasonable and follow implausible paths that sometimes lead to nothing, sometimes there are good ideas that clinicians have not thought of as they keep their nose to the grindstone.” (Clinician-research, university hospital, 16 years of experience in cancer research)
Views of researchers on evaluation of translational cancer research and indicators (qualitative study)
Existing evaluation systems reward translational research less favourably
“We expect translational research to have much more impact on treatments and patients care, all medical aspects that fundamental research… OK there can be a researcher that spends a lot of time, even all his life, researching on things that will never be applied. Translational research is applied research. Which means there are specific evaluation criterions to this applied research. In particular the impact in terms of health that this translational research should have.” (Clinician-researcher, comprehensive cancer centre, 25 years of experience in cancer research)
Classical indicators are acceptable but not sufficient
“Sometimes a translational research project is a project that does not confirm a data. […] We talked a lot about MET for head and neck cancer, we wanted to explore this marker and see if we could offer treatment against this oncogene. […] We came to the conclusion that there was no MET abnormality and it was not interesting to develop clinical strategies for that. We had a lot of trouble publishing those data because they were negative.” (Engineer, comprehensive cancer centre, 16 years of experience in cancer research)
“The only difference between a journal with a high impact factor and a journal with an average impact factor is the quantity of data that you add to strengthen you hypothesis. But if your hypothesis is strong with a small number of figures, you don’t need to spend two additional years on the same hypothesis.” (Clinician-researcher, comprehensive cancer centre, 7 years of experience in cancer research)
“This indicator [patent count] is pretty good because it measures innovation and clinical applications. But it has negative consequences because it can discourage collaboration. It can incite researchers to keep their biological material and not share it.” (Pharmacologist, research agency, 26 years of experience in cancer research)
Measuring translational research in terms of clinical applications or patient outcomes
“Let’s take the example of ERCC1, which produced two ‘New England Journal of Medicine’, one therapeutic trial and eventually had not clinical application. In your criteria, it comes off at the top of the list, there is a patent on it… it ticks all the boxes. But there was no clinical application; it is something that was dropped. […] In practice not only it has not been developed, but on top of that there has been potentially a loss of chance since we used a biomarker that was retrospectively invalidated.” (Clinician, comprehensive cancer centre, 9 years of experience in cancer research)
Suggestion or comment by participant
|
Possible indicator (according to author)
|
Authors’ comments
|
---|---|---|
“Collaboration between biologists and epidemiologists is important and should be measured in terms of outputs, such as joint papers” | Number of publications co-authored by an epidemiologist and a biologist | This is the only indicator of multidisciplinarity proposed by a participant; however, it is very specific to research in molecular epidemiology |
No similar indicator has been created; indicator added to the Delphi survey | ||
“One interesting indicator would be the number of patients in a clinical trial benefiting from a biomarker identification” | Number of patients included in a clinical trial with a biomarker identification | That indicator would be studied by a survey of cancer centres; indicator added to the Delphi survey |
“The point of translational research is to transfer to clinical practice. So it is supposed to generate clinical studies. Ideally it [an evaluation measure] would be how many positive studies had been generated” | Number of hypotheses generated | Literature suggests one indicator of ‘number of hypotheses generated’ [6], but no methodology is proposed; indicator added to the Delphi survey |
“A good indicator of translational research would be its capacity to generate hypothesis to test in the clinic. […] So the protocols of clinical validation that have been generated” | ||
“We should ensure whether the tools developed are effective enough to process data the correct way” | Measures of effectiveness of developed tools | The participant clearly specified that it was an indicator specific to their field (bioinformatics) and not applicable to whole translational research; not added to the questionnaire due to lack of clear definition |
“The primary aim [of translational research] would be to adapt technologies to the general population. So it should be evaluated on this aspect” | Use of developed technologies in practice | No existing indicator; not added to the questionnaire due to lack of clear definition |
“ Developing a biomarker in translational research will help to select patients that will benefit from a treatment, that is a real proxy of translational research efficacy” | Number of biomarkers developed | Literature suggests one indicator of ‘number of biomarkers identified’ [6], but no methodology is proposed; indicator already part of the Delphi survey |
“What should be measured, for translational research, is the benefit for the patient. Not the final benefit […] but the interim benefit, such as biomarkers developed” | ||
“The ideal for translational research, it that it modifies patient care. So that can be a long-term objective, but […] if there are interim step” | ||
“Ideally, a translational study should lead to an application, which means, from clinical to basic research, to a fundamental research project, and in the opposite direction, to a clinical application, such as a clinical trial, the validation of a biomarker, or an imaging study” | ||
“[translational research should be evaluated] in terms of publications and implementations in the clinics. […] Also guidelines” | Clinical guidelines generated | There are two existing indicators measuring the transfer of research in clinical guidelines: number of clinical guidelines generated and citation of research in clinical guidelines; indicators already part of the Delphi survey |
“The development of database is also an important structural factor… an indicator” | Number of databases created | Literature suggests one indicator of ‘number of databases created’ [6], but no methodology is proposed; indicator added to the Delphi survey |
Selection of indicators (survey)
First round survey
Round 1
|
Round 2
| ||||
---|---|---|---|---|---|
Indicator
|
Feasibility
|
Validity
|
Status
|
Validity
|
Status
|
Number of clinical trials | 81% | 84% | Included | – | Included |
Percentage of patients included in a clinical trial | 77% | 81% | Included | – | Included |
Number of peer-reviewed publications | 95% | 81% | Included | – | Included |
Number of citations | 88% | 85% | Included | – | Included |
Number of public-private partnerships | 75% | 78% | Included | – | Included |
Impact factor | 96% | 76% | Included | – | Included |
Institutional h-index | 92% | 73% | Second round | 83% | Included |
Number of publications co-authored with another organisation | 92% | 72% | Second round | 76% | Included |
Mean number of citations per article | 91% | 69% | Second round | 66% | Included |
Number of highly cited publications | 78% | 67% | Second round | 69% | Included |
Number of publications in top-ranked journals | 85% | 65% | Second round | 69% | Included |
z-index | 83% | 71% | Second round | 69% | Included |
Number of publications with international collaboration | 92% | 67% | Second round | 85% | Included |
Number of biomarkers identified | 33% | 52% | Modified | 58% | To be discussed |
Citation of research in clinical guidelines | 71% | 71% | Second round | 62% | To be discussed |
Generation of clinical guidelines | 74% | 67% | Second round | 54% | To be discussed |
Citation of research in public health guidelines | 60% | 58% | Modified | 50% | To be discussed |
Number of patents | 88% | 65% | Second round | 38% | Excluded |
Number of patients in a clinical trial with a biomarker identification | 56% | 58% | Excluded | – | – |
Number of biological samples collected | 69% | 50% | Excluded | – | – |
Number of biological samples transmitted | 42% | 34% | Excluded | – | – |
Number of hypotheses generated | 46% | 46% | Excluded | – | – |
Number of diagnostic tests created | 50% | 45% | Excluded | – | – |
Number of database generated | 50% | 38% | Excluded | – | – |
Number of research projects ongoing | 57% | 29% | Excluded | – | – |
Number of assays developed | 0% | 0% | Excluded | – | – |
Number of visits to EXPASY server | 75% | 0% | Excluded | – | – |
Clinicians’ awareness of research results | 22% | 36% | Excluded | – | – |
Changes in clinical practices | 21% | 43% | Excluded | – | – |
Contribution to reports informing policy makers | 58% | 35% | Excluded | – | – |
Number of presentations at key selected conferences | 60% | 56% | Excluded | – | – |
Citation in medical education books | 42% | 48% | Excluded | – | – |
Number of conferences held | 71% | 38% | Excluded | – | – |
Citation of research in the media | 41% | 28% | Excluded | – | – |
Number of papers co-authored with the industry | 74% | 61% | Excluded | – | |
Citation of research in patents | 79% | 58% | Excluded | – | – |
Patent h-index | 52% | 52% | Excluded | – | – |
Number of patent citations | 73% | 48% | Excluded | – | – |
Number of spin-off companies created | 86% | 43% | Excluded | – | – |
Partnership-ability index | 82% | 50% | Excluded | – | – |
Dependence degree (d-index) | 67% | 50% | Excluded | – | – |
Proportion of long-distance collaborative publications | 43% | 0% | Excluded | – | – |
Number of publications co-authored by an epidemiologist and biologist | 52% | 41% | Excluded | – | – |
Age-weighted citation rate | 71% | 33% | Excluded | – | – |
b-index | 83% | 43% | Excluded | – | – |
Central index | 60% | 50% | Excluded | – | – |
CH-index | 50% | 0% | Excluded | – | – |
Crown indicator | 33% | 0% | Excluded | – | – |
e-index | 80% | 50% | Excluded | – | – |
g-index | 100% | 0% | Excluded | – | – |
Hg-index | 33% | 0% | Excluded | – | – |
j-index | 75% | 33% | Excluded | – | – |
m-index | 50% | 25% | Excluded | – | – |
m-quotient | 67% | 60% | Excluded | – | – |
Mean normalised citation score | 50% | 33% | Excluded | – | – |
Q2 index | 25% | 0% | Excluded | – | – |
r-index | 83% | 33% | Excluded | – | – |
SP-index | 100% | 50% | Excluded | – | – |
w-index | 71% | 33% | Excluded | – | – |
x-index | 50% | 0% | Excluded | – | – |
Second round survey
Physical meeting
Indicator
|
Definition
|
Type of indicator
|
Why it measures the impact of translational research
|
---|---|---|---|
Number of clinical trial | Number of clinical trials active in a cancer centre in a year | Process | Those indicators measures how research and care are integrated into a hospital |
Percentage of patients included in a clinical trial | Ratio of the number of patients included in a clinical trial to the total number of patients | Process | |
Number of peer-reviewed publications | Number of peer-reviewed publications authored by the institution | Output | Peer-reviewed publications play a fundamental role in the circulation and exploitation of knowledge |
Number of citations | Number of citations of the published articles of an institution received within a time span | Output | It measures how an article has influenced other scientists and its impact on the advancement of knowledge |
Impact factor | The ratio of the number of citations to the number of citable items of a journal | Output | This indicator measures the visibility of the production of a research institute |
Institutional h-index | Indicator that combines the number of articles produced by research units and their number of citations | Output | As this indicator combines metrics of quantity and visibility, it measures the possible influence of the entire production of a research institute |
Number of public-private partnerships | Number of partnerships between an academic research centre and the industry | Process | Public-private partnerships facilitate the translation of research finding into clinical applications |
Mean number of citations per article | A ratio of the number of citations received by an institution to their number of publications | Output | This indicator allows a comparison of the potential influence of an institution adjusted for their age |
Number of highly cited publications | Number of published articles with a citation count above a certain threshold | Output | This indicator potentially measures the number of articles that had a high impact in the research community |
Number of publications in top-ranked journals | Number of publications in journals with the highest impact factor in the discipline | Output | This indicator allows adjustment for differences in citation practices between disciplines |
z-index | An indicator combining the number of publications of an institution and the impact factor of those publications | Output | As this indicator combines metrics of quantity and visibility, it measures the possible influence of the entire production of a research institute |
Number of publications co-authored with another organisation | Number of publications co-authored by researchers affiliated to another research institution | Output | Cooperation benefits research by bringing new ideas and methods and helping to reach comprehensive expertise. In cancer research, collaboration between institutions is particularly crucial for research on rare cancers where it can be challenging to include enough patients. Those indicators measure the proportion of research performed during a collaboration |
Number of publications with international collaboration | Number of publications co-authored by researchers affiliated to a research institution in another country | Output | |
Number of biomarkers identified | Number of valid biomarkers identified by the research institution and published in a peer-review journal. | Outcome | Biomarkers play a fundamental role in developing personalised treatments and possibly improving patient outcomes |
Generation of clinical guidelines | Number of clinical guidelines authored by the institution | Outcome | Clinical guidelines facilitate the adoption of research findings into practices and aim to improve care quality Those indicators measure the possible influence of research on the improvement of clinical practices |
Citation of research in clinical guidelines | Number of articles cited in clinical guidelines | Outcome | |
Citation of research in public health guidelines | Number of articles cited in public health guidelines | Outcome | This indicator measures the possible influence of research findings on public policies (e.g. cancer screening) |