Methods
On the basis of the response rate and comments, the Delphi method and the online survey approach were generally well received. Experts were willing to devote time and effort to complete two online surveys over five months and also provided helpful comments. Several researchers commented that it was good to have the opportunity to participate in a transparent approach to priority setting. The literature recommends anything from 10 to 80 experts on a Delphi panel [
9,
11] and the study response rates compare favourably with other recent online Delphi surveys. For instance, an expert online Delphi survey on research and development into drugs for neglected diseases reported first and second round response rates of 30% (117/388) and 14% (56/388), respectively [
14]. The sample selection was purposive in that it identified interested experts from specific sources, such as global malaria initiatives, review committees, conference presentations, and relevant publications. There was no notable difference in demographic characteristics of participants between the two rounds. Both the first and second round samples were biased towards those working in academia and more than half stated their place of residence in a high-income country. As with other Delphi studies, it is impossible to judge how and if the results may have differed if the panel characteristics were different [
14]. There was some attrition in the response rates over the two rounds, with 11 participants dropping out between the first and second survey. However, as the characteristics of those completing both surveys were broadly similar it is unlikely that drops-outs overly influenced the nature of responses.
There are several strengths in the Delphi method as a structured process for gathering opinions. First, it enables input from individuals using a systematic anonymous and iterative approach to gain consensus on an area of interest. This is likely to allow a more inclusive process of determining values than focus group discussions [
10]. Delphi surveys also allows for a range of individuals to express their opinion which can then be reassessed by considering the input from other participants, with the eventual aim of reaching some convergence [
15]. A significant advantage of the Delphi is that participants are very much aware at each stage of the results of the previous survey rounds, and there is scope for each panel member to provide more detailed feedback on both the process and the results.
While the Delphi method has several appealing characteristics, there are limitations. First, the approach, as noted earlier, is not designed to be statistically representative. Therefore, it is not possible to generalize these findings to the population of malaria research experts or researchers in general. Other more quantitative techniques, such as discrete choice experiments, offer a way to systematically assess how groups value different types of research outcomes, although they are time consuming, require a lot of data and can be expensive to undertake. A second drawback of the Delphi approach is it can be a lengthy process due to its iterative nature. Linked to that, the multiple feedback processes, integral to the Delphi concept, mean the problem of low response rates are magnified. If a certain group of experts discontinue their responses during various stages of the Delphi process, the quality and reliability of the information obtained could be threatened [
10]. A potentially more serious issue is that the iteration characteristics of the Delphi technique can potentially enable investigators to ‘mould’ opinions. One experiment conducted in the 1970s by Scheibe and colleagues showed that Delphi subjects would rate their responses differently after receiving distorted feedback [
24]. So-called ‘acquiescence’ bias is a particular risk when asking respondents whether they agree with the presented findings from the first round.
A further drawback is an assumption that Delphi participants “are equivalent in knowledge and experience” [
25]. There are good reasons for concern that the knowledge of participants could be unevenly distributed, whereby some participants have much more expertise in certain areas than others. Experts who have less in-depth knowledge of certain topics may be unable to distinguish the most important statements that have been identified by those subjects who possess wider knowledge. In these instances the outcomes of a Delphi study could be the result of identifying a series of ‘general’ issues rather than an in-depth exploration of the topic. While this is certainly a risk, it is arguably a risk for all qualitative investigations that seek to gather some kind of group consensus. None of the issues or questions raised by the Delphi were thought to beyond the expertise of any of the participating experts.
Another potential criticism of the study approach is that the final sample was biased towards research experts largely working in academic institutions. It is acknowledged that malaria research, particularly operational research, is also conducted outside of traditional academic settings, such as in non-governmental organizations or other implementing agencies. It could be the case that a sample with a more diverse background could have led to different results, such as an even higher weighting on later stage impact. Future work could explore the views of those working in other settings.
The dominance by northern academics based in high-income countries means there may be a bias in the round 1 free text mentions that was propagated to round 2, given the structure of the iterative process. On reflection more information on the areas of particular interest of panel members could have been requested. However, it might have been hard for experts to narrowly define their interests. It is also likely that many panel members had overlapping interests, making interpretation difficult.
Research priorities
The study highlighted the importance of past research developments in malaria, particularly the development of new drugs, insecticide-treated bed nets and broader vector control. However, experts were also keen to highlight that increased funding for malaria interventions more generally were critical to improvements in the scientific knowledge base. Priorities for future malaria research investment reflected areas where the most success had been seen in the past (vector control and new drugs), but also highlighted the importance of tackling emerging drug and insecticide resistance, in particular resistance to artemisinin-based combination therapy. Experts agreed that staying ahead of the challenge of resistance requires expanding the pipeline of new drugs, diagnostics and other tools. One participant commented that there has been a lack of “true innovation” in the community, arguing that “our current best tools are still also our oldest”. Local capacity building and wider health systems and operational research were also highlighted. This possibly reflects a more general concern that there is a potential ‘disconnect’ between researchers and implementers and there are insufficient people with the skills to bridge this gap. One expert commented that funders are more willing to fund the clinical science with less supporting operational or implementation research.
Looking beyond malaria, experts identified tackling broader drug resistance (i.e., antimicrobial and antibacterial) as the most important area for attention in the next 20–50 years. This reflects wider political and global concern on the growing threat of resistance; both the UK and WHO have raised this issue as a global threat [
26,
27]. The one case where there was apparently less congruence between the two rounds was on the importance of NCDs. NCDs was the most frequently mentioned future global issue in the first round, but participants ranked NCDs lower (seventh) in the second round when it was presented alongside other issues. Most obviously, this may reflect the disciplinary bias of the group, who all worked in infectious disease. But there is little doubt that chronic NCDs are reaching epidemic proportions worldwide [
28]. NCDs currently make up more than 40% of the disease burden and are expected to overtake communicable diseases as the major cause of mortality in low-income countries (LICs) by 2030 [
29]. Indeed, the emergence of NCDs in LICs is partially the result of progress in tackling high-burden communicable diseases such as HIV/AIDs, tuberculosis and malaria with the resultant increase in average life expectancy in many countries [
30]. However (to participants at least) it may be that the perception of the NCD burden in sub-Saharan Africa is still overshadowed by the unfinished agenda on communicable diseases as well as tackling emerging infectious diseases of epidemic potential, such as Ebola.
Valuing research outcomes
A key aim of the Delphi survey was to elicit expert views on the relative importance of different global health research outcomes. In this context ‘value’ is equated to percentage weight assigned to each outcome category. The findings suggest that malaria research experts prioritized research outcomes that focused most on ‘health and health sector benefits’ and ‘informing policy and product development’. The overall rankings did not change over the two rounds of the survey, although there were some differences within each round when the results were broken down by experience, region and employment sector. ‘Contribution to scientific knowledge’ came lower down the priority list, suggesting that participating experts valued efforts to move research discoveries to health products and services over pure advances in scientific knowledge. Broader economic outcomes were valued the least important. The finding that economic outcomes are weighted less highly than health outcomes is perhaps not that surprising. This is a similar finding to a previous study looking at the outcomes of medical research in Canada [
8]. The authors suggest this could reflect researcher views on the limited potential of research to generate pure economic returns, but it could also reflect the lack of data on the economic returns of health research more generally, malaria research in particular. Similarly, a more recent study conducted in the UK suggested that: “achieving higher life expectancy for adults living with a common chronic disease in the UK is one of the highest priorities for the general public and researchers—well ahead of commercial and employment benefits (from the research)” [
31]. That said, the same study found that researchers and the general public agreed that creating substantial numbers of jobs through research is important, suggesting that perceptions of economic impact can be more nuanced. Exploring the economic rates of return from global health research will be the subject of future work.
It is worth keeping in mind that the different outcomes from research are not mutually exclusive, and are often interlinked in a logic chain from a research award or grant, through to research outputs through to impact. A research programme that performs well on one dimension, for example building research capacity, may mean performing less well on another dimension, such as publishing high-impact journal articles. Nonetheless, the findings suggest that more value should be placed on later stage impact indicators with participants giving a combined weight of 36% to the two indicators (i.e., health and health sector benefits + broader economic benefits). This is higher than the impact weight of 20% assigned by the UK Research Excellence Framework [
32]. A key question, inevitably, is how to accurately identify, attribute and measure those impacts.