What is research and what is its value?
There are variances in the nature of research and thus its meaning, purpose and value. There is agreement nevertheless, that the primary intent of research is to systematically generate new knowledge. To unpack its meaning, we looked at a definition of research to which Australia’s NHMRC submits and that originated from the United Kingdom Research Assessment Exercise. It proposes that research “… includes work of direct relevance to the needs of commerce, industry, and to the public sectors; scholarship; the invention and generation of ideas; images, performances, artefacts or design where these lead to new or substantially improved insights; and the use of existing knowledge in experimental development to produce new or substantially improved materials, devices, products, and processes including design and construction” [
15]. Inferences drawn from this definition indicate that whatever form research takes, it must be useful and directly relevant to the advancement of knowledge for societal progress and human health and flourishing; and designed to produce innovations that have impact in such contexts. For these reasons, research is necessarily intertwined with: 1) societal ethics, whereby research benefits must outweigh any harm; and 2) values that are temporally and contextually open to interpretation and change.
Despite variance in perceptions of research, the value of health research to individuals and society is indisputable [
2]. But it should be noted that research is a means to an end, rather than an end itself; and that alone it will not necessarily derive benefits or be of value. But the generation of new knowledge contributes to improvements in health care and public health by providing information about disease trends, risk and protective factors, patterns of care and health care costs, and developing new therapies and treatments, and assessing the effectiveness of health interventions. Research alone does not generate benefits. The degree to which health research is valuable and makes substantive contributions to society however, depends on its nature, quantity and quality [
16] and whether action processes or translational work occurs across the life, or at the back-end of research [
17]. In terms of the latter, it is clear that “society can reap the benefits of successful research studies only if the results are converted into marketable and consumable products (e.g., medicaments, diagnostic tools, machines, and devices) or services” [
18]. But achieving research benefit involves time-oriented processes; for instance, there is reportedly “an average time-lag between research funding and impacts on health provision of around 17 years” [
19]; for example, the translation of a research discovery in a biomedical lab to day-to-day clinical practice [
20].
What is research benefit and how is it demonstrated?
In a policy context, Indigenous Australian people define research benefit as “the establishment or enhancement of capacities, opportunities or outcomes that advance the interests of Aboriginal and Torres Strait Islander peoples and that are valued by them” [
1]. Here, benefit is demonstrated by reciprocity [
1]; where reciprocity refers to the Indigenous value of mutual obligation and which aims “to achieve an equitable distribution of resources, responsibility and capacity and to achieve cohesion and survival of the social order” [
1]. Translation of reciprocity into a research context infers “inclusion, recognition of partners’ contributions, and ensuring that research outcomes include equitable benefits … for communities or individuals” [
1]. For researchers, reciprocity requires honouring research partnerships by demonstrating benefit; as defined and valued by the community according to their priorities and which contributes to their cohesion and survival [
1].
While capturing the meaning of research benefit necessarily takes the form of ethical value-oriented methodological considerations in the decision-making processes of research endeavours, there are also conceptual issues to be resolved in distinguishing the difference between the terms that speak to benefit. For example, in using the terms ‘research impact’ and ‘research benefit’, some people use benefit and impact as discrete terms; some use the two terms interchangeably; and different disciplinary perspectives direct various usage. In this article, the authors worked from the assumption that the two terms are distinct from each other, while at times interrelated and interdependent. ‘Research impact’ is considered to be any area of influence flowing from the research endeavour including those that flow from research processes; these can be both positive and negative. ‘Research benefit’ on the other hand, flows from areas of impact. This benefit can be intended or unintended, inside or outside the immediate research environment, direct or indirect, tangible or intangible, immediate, short-term or longer-term; but benefit must be positively oriented and represented as elements of value derived from research.
There are apparent inconsistencies regarding the meaning of research impact and benefit, and therefore also disagreement about how it can be measured. The Australian Research Council (ARC) [
21] defines research impact as “the demonstrable contribution that research makes to the economy, society, culture, national security, public policy or services, health, the environment, or quality of life, beyond contributions to academia”. It includes, but is not limited to, an effect on, change or benefit to: “the activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding of an audience, beneficiary, community, constituency, organisation or individuals in any geographic location whether locally, regionally, nationally or internationally”. A group from the United Kingdom who have been particularly active in the field of research impact, the London School of Economics (LSE) Public Policy Group [
22] assert that it is “an occasion of influence and hence it is not the same thing as a change in outputs or activities as a result of that influence, still less a change in social outcomes”. They suggest two broad areas of impact from which benefits are derived: 1) Academic impacts - influences upon actors in academia or universities e.g., as measured by citations in other academic works; and 2) External impacts - influences on actors outside higher education, in business, government or civil society e.g., as measured by references in government documents. Other UK researchers have expanded on those areas of impact. For instance, Kuruvilla, Mays, Pleasant & Walt [
23] present an evidence-based assessment of the nature of impact and its associated benefits. In developing their health research impact framework, based on a literature review and empirical analysis of selected London School of Hygiene and Tropical Medicine research projects, they identified four key areas of research impact: 1) research-related, 2) policy, 3) services: health and intersectoral, and 4) societal impacts. They claim that the four categories are not exhaustive or mutually-exclusive, and are at times interdependent and overlapping. Yet others have suggested that these elements too often operate in silos [
24] or have a stronger focus on research-related impact and benefits which largely influence actors in academia or universities and thus often represent the key benefits measured [
23]. For example, Excellence in Research for Australia has been introduced by the Commonwealth Government to provide more rigorous measures of research quality across the higher education system [
21]. It aims to assess research quality using a combination of metrics focused on researchers, research outputs, research income, esteem and applied measures. These indicators of productivity are quantifiable, but generally do not translate into robust measures as valued by end-users of research. We take the view that academic benefits and other contributions need not be mutually exclusive and that the various elements should concurrently interact.
The debate around research impact and benefit nevertheless, is ever-progressing and reflected in recent planning by the Commonwealth Government of Australia [
25]. The government aim is to develop a mechanism by which the broader economic, social and environmental benefits resulting from government research investment, including the benefits arising from university-based research can be assessed [
25]. Driving this change is an attempt to broaden the scope of research impact and benefit because although the benefits from research have been under scrutiny since the 1990s [
26], the primary focus was on benefits flowing from the impacts of research on academia and scientific knowledge [
17].
Despite major developments driving changes in Indigenous health research methodologies and practices in recent decades – like the establishment of the Lowitja Institute, Australia’s only Indigenous-controlled health research organisation [
4] and the Aboriginal Health and Medical Research Council of New South Wales - Indigenous people are still concerned about the ethics of research, including how research is developed, done and whether it has tangible benefits as determined them [
27]. In leading reform, such institutions have applied a decolonising lens to think about, and implement broader Indigenous research strategies that foreground Indigenous knowledge, values and practices [
28]. A decolonising approach requires a power shift in research relations and is embedded in a social justice perspective. It entails for instance, the privileging of Indigenous voices and epistemologies in collaborative research endeavours. For example, Lowitja Institute has embraced collaborative research approaches and lead the way in funding knowledge exchange and translation that ensures research is priority-driven and that evidence-based research findings are used to improve Indigenous health practice and well-being in sustainable ways [
29]. The Institute has also actively pursued identifying the future demands for Indigenous health research to ensure the relevance of research into the future and thus maximise potential benefits.
Indigenous research reform has however, focussed predominantly on process issues; and only in more recent years has research translation to maximise benefits become a priority. As an example, the reporting of research benefit and research translation in Indigenous health is miniscule in Australia. A systematic search of four key databases (Medline, Scopus, Informit and Cinahl) using the terms ‘health’, ‘research benefit’, ‘research impact’, ‘Aborigin*’, ‘Indigenous’, ‘research translation’ and ‘knowledge translation’ and with the limits of Australia and 1995 – 2014 returned a total of 59 hits. Of these, only 2 publications considered, but did not focus on, research benefits and translation.
Indigenous researchers and scholars however, have long sought to reform mainstream Indigenous health research methodologies and practices [
28,
30‐
32]. With the goal of achieving measurable benefits as defined by Indigenous people, they have led these reform agendas and created Indigenous methodologies that have as one of their key defining features sustainable benefit. These reform agendas align with Indigenous aspirations of empowerment and self-determination. As such they have an emancipatory imperative, political integrity and privilege Indigenous voices [
30] by recognising Indigenous ways of knowing, ways of being and ways of doing [
31,
32]. Simultaneously, funding bodies such as the NHMRC [
1] have produced ethical guidelines for Indigenous research and prioritised research funding based on ethical community engagement and research relevance, that is identified and driven by Indigenous communities. In response, many non-Indigenous researchers have amended their approaches to research by nurturing closer partnerships with Indigenous research stakeholders and reoriented to participatory action research methodologies to increase the appropriate application of research evidence in policy and practice [
11,
33,
34]. Funding bodies have also revised their practice; for instance, the CEO of Australia’s NHMRC advises that the assessment and ranking of funding applications are also determined based on the importance of an idea as judged according to its ability for impact on knowledge development and beyond [
35].
Apart from the call from Indigenous people to increase research benefits, in the broader policy context of research there has been a demand for more accountability in public spending across all sectors. In times of tight fiscal government environments, governments and others are increasingly shifting toward research-based evidence as a way of maximising efficiency and efficacy in policy and public service delivery. Nevertheless, there has been increased health research expenditure. In Australia and most OECD countries, expenditure on health care, both public and private, has been rising faster than Gross Domestic Product (GDP) over the past decade [
36]. In Australia, research and development expenditure in health as a proportion of total spending on other health care increased from 11.0 % in 2001–2002 to 17.5 % in 2011–2012 [
36]. For example, the Australian Government invested in health research with expenditure increasing from $783 million in 2009–10 to $798 million in 2011–12 [
36]. Increased expenditure means a justifiable expectation of increased returns on investment. The role of publicly-funded research organisations, as well as public funding arrangements for research has been redeveloped to reflect this position [
21]. That is, research must be now focused to “maximise resource effectiveness, quality and impact and its ability to meet Australia’s strategic needs” [
21].
Additional to these reforms, Thomas, Bainbridge & Tsey [
2] more recently suggested that health equity for Indigenous Australians is also more likely to be realised through “improved research processes and ethical frameworks to guide and keep accountable researchers and research organisations, alongside the development of methods to assess research impact and benefit for Indigenous people”. Others have also explored these issues. For instance, research implementation fidelity has been raised as an concern [
37,
38] and Humphrey [
11] has raised questions about the extent of Indigenous research transformation and its potential to bring “about fundamental change in research practice” vis-à-vis “being sidelined into too great a reliance on written guidelines and positive rhetoric”.
However, and despite evidence of changing trends in Indigenous health research, there has been little focus on ways of systematically determining whether a particular piece of research is of benefit to Indigenous people. Indeed, there “is no systematic process of measuring the broader economic, social and environmental benefits of [any] publicly funded research” at all [
25]. The Australian Government however, is focussed on improving accountability and the benefits arising from university-based research and is working toward building a framework to measure research benefit in a systematic way [
25]; this is not surprising given that only one-tenth of research funding leads to measurable impact and benefits [
17]. For example, a survey of 76 research and development managers in United States firms were asked to make estimations of the proportion of products or processes that could not have been developed in the last 10 years without academic research. Calculations were made at only 10 % for new products and 11 % for new processes [
17]. Herein, lays further evidence to compel researchers toward working to achieve a more coordinated, coherent approach that maximises benefits from funding investment in research.
Whose perspective?
The preceding definitions demonstrated the variable contexts in which people distil the meaning of research benefit and the hefty weighting that academic impact and related benefits hold. The delineation of benefit and its explicit priority setting is an important first step in its demonstration, achievement and measurement; and inevitably leads to questions about benefit from whose perspective. In contemplating questions of research benefit for Indigenous Australians, a number of circumstances come to mind that need to be well thought through. One outstanding issue to be considered when articulating an impact-benefit narrative is that although there are overlapping aspects, the concept of research benefit does not translate equally across Indigenous and non-Indigenous contexts or the different levels at which impact is manifest. Traditionally, impact and therefore benefits has been measured by peer accountability in academia, as opposed to social accountability, with academic productivity measures such as research income, bibliometrics and citations taking precedence.
As researchers, we are very familiar with grant proposals where benefits are expressed in terms of research users, beneficiaries, communication and expected impact; research project evaluations and reports that fundamentally provide accountability for funding investment; media information, research dissemination; estimations of research impact on policy and practice; and individual or institutional research impact submissions [
23]. However, the research impact debate must take account, for instance, of the various standards of accountability (to whom), impact priorities (for whom), positive and negative impacts, and biases that operate in describing impact and measuring benefit [
23].
Understanding how people value and interpret the benefits of research is critical. While academics talk about research impact, Indigenous people talk about ethical positions and ‘tangible’ benefits such as responses and solutions to issues studied in research projects that are meaningful to their lives. These benefits might include the development of historical artefacts and languages that preserve culture and benefit future generations and direct benefits such as the translation of any research knowledge into more immediate relevant and consumable products. However, tangible for Indigenous people can mean both tangible and intangible benefits when perceived from Western standpoints; intangibles might include cultural determinants and concepts of wellbeing such as identity, empowerment and control. The question posed is therefore: to what extent do current measures fulfil the articulation of benefit from the perspective of Indigenous people and organisations with whom researchers work; and how do we work toward mutual research outcomes? – as the NHMRC Chief Executive Officer states, benefit demonstrated in the academic context, while valuable, should be seen as “just a milestone on the road to real impact in improved health” [
35].
What approaches have researchers taken to achieve impact and increase benefit?
Research ethics guidelines such as those expounded by the NHMRC and other international ethics review boards emphasise community-driven and controlled participatory approaches to conducting research with Indigenous partners. However, without authentic uptake and implementation of these, and other collaborative research processes; along with contemplated action at the back-end of research (research translation), research effects are minimised. Silburn et al. [
39] note the development of the Cooperative Research Centre for Aboriginal Health’s facilitated research development process as a mediating approach for improving ethical implementation of research processes and thus increase benefit. They advocate that changing the processes of research accordingly – from priority setting to research transfer is a necessary part of maximising research benefit for Indigenous people. In particular, they suggest completely re-modelling research processes to improve the likelihood of research use. These included: 1) Indigenous people becoming direct and active participants; 2) identifying and prioritising issues of relevance to Indigenous people; 3) incorporating Indigenous knowledges and perspectives in processes and findings; 4) reporting findings were meaningful; and 5) engaging all potential end-users from the outset.
Partnership approaches are amenable to achieving impact and benefits in Indigenous research contexts. This is because partnerships strengthen the relationship between researchers and research users and bring together different knowledges and expertise to accomplish more than what could be achieved alone; thus improving the prospect of benefit [
34]. Importantly, adding reflective processes that encourage the development of shared understandings of the situation under study also invite opportunities to examine expectations and interpretations of benefit. Action research is favoured for its usefulness in implementing action and change throughout the research process. Action grounded in localised knowledge and implemented throughout the research process to maximise impact and refine planning for instance, was a priority for Yarrabah stakeholders in developing a community social and emotional wellbeing action plan [
40]. Building Indigenous research capacity, another strategy implemented by the NHMRC, has been successful in increasing the numbers of Indigenous researchers through its population health capacity building grants [
41]. These grants were introduced in response to a review of Australian public health research and designed to strengthen research workforce capacity that has demonstrated a potential for the delivery of best practices; in turn potentially contributing to better engagement and research outcomes [
2]. However in contemplating capacity development efforts, again, the foci often reside in the academic arena while Indigenous lay knowledge and the ways in which it can assign meaning and inform action to reduce health inequities is often overlooked. Working at the interface of scientific and cultural knowledge is an important undertaking in striving for change and maximising benefit [
42].
These developments speak to the implementation of new research. Increasingly, alternative approaches such as systematic reviews are being conducted to ascertain the effectiveness of Indigenous health programs without burdening Indigenous populations with more research. Through the conduct of reviews, concerns have been expressed about the over-emphasis on descriptive research rather than research evaluating interventions in Indigenous health [
16,
37,
43‐
48]. These reviews have additionally identified what Paul et al. [
43] describe as “the sorry state of the evidence base for improving the health of Indigenous populations”. Further to identifying the dominance of descriptive research, many reviews note the overwhelmingly paltry numbers and poor methodological quality of studies. Given that the degree to which health research is valuable depends on the nature, quantity and quality and fitness to purpose of available evidence [
16]; more robust research designs are needed to improve outcomes. Journal editors too have noted this gap; for instance, the
MJA’s editors flagged that Indigenous health research has been too “observational and deficit-focused, with a dearth of interventional studies” and have “even considered putting a moratorium on publishing the many observational studies” being submitted [
49].
How can research benefit be measured?
Defining research impact and benefits is difficult, but drawing a linear link between a particular research study and subsequent improvement in health, and measuring that impact, is an even more intrinsically difficult task [
35,
50]. This situation is hampered at times by political processes that can occur between research conduct and the realisation of outcomes, and over which researchers might or might not have control. Assessing the benefits of research is a necessary activity and can be used for a number of different purposes; thus approaches to interpreting benefit can be made by measuring it both quantitatively and qualitatively, including economic evaluations. However, narrative (qualitative) approaches have been critiqued because they are subjective, expensive to implement and hard to compare across contexts. Nevertheless, what we measure is based on what we value most; and perhaps why the motivations of the dominant society toward academic measures have been prioritised. Research findings and thus benefits are filtered through people’s interests and beliefs [
51]; these need to be negotiated and clearly defined for research stakeholders, accounted for from the outset of the research and aligned with the objectives and intended outcomes. Benefits also need to be reviewed progressively. The question thus becomes what kinds of benefits are expected, rather than whether researchers can create impact through a plausible narrative. In doing that, whose voices are heard in contemplating impact needs to be considered, as is the need to challenge the assumptions of researchers and others.
The current state of Indigenous research assessment embodies a Western cultural framework based on measurable units and quantifiable methods and indicators. As a result, biomedical and economic indicators of research assessment do not account for measurable aspects of community living and wellbeing that are not easily measured. These less easily measured aspects include intangible cultural heritage and wellbeing, Indigenous people’s worldviews, associations and relationships [
52,
53]. An easily accessible example is the effects of research on the health and wellbeing of Indigenous people. But the concept of health and wellbeing for Indigenous people expands on the Western concept of health to also include socially and culturally-based understandings of life. According to the National Aboriginal Health Strategy [
9], Indigenous health is not just the physical wellbeing of an individual, but the social, emotional, and cultural wellbeing of the whole community in which each individual is able to achieve their full potential [
53,
54]. Thus health for Indigenous people is a multi-dimensional concept. It embraces all aspects of living, including those valued by Western societies such as the social determinants of health, while also stressing the importance of survival and flourishing in harmony with the environment.
While the reference to Indigenous wellbeing indicators in policy is made, Indigenous cultural heritage as a factor in the wellbeing of Indigenous people is not simultaneously recognised [
53]. Instead, the foci is on: life expectancy as a measure of health and wellbeing; employment and income, opportunities for self-development, living standards and self-esteem as important for overall wellbeing; household and individual income as a determinant of economic and overall wellbeing; home ownership as an important element in improving Indigenous wellbeing; infant mortality as an indicator of the general health and wellbeing [
53]. These indicators of health and economic standard of living for Indigenous people have no reference to Indigenous wellbeing as it is understood in international and local Indigenous contexts [
54]. The exclusion of intangibles such as wellbeing occurs primarily because the concept is not valued by Western systems in the same way and is difficult to measure using standard scientific approaches. However further to that, it should be noted that such societal impact approaches are evolving and more research is required to develop the appropriate methods. Also required is political will on the part of policy makers to invest in developing the methods to quantify what is important rather than what is easy to quantify. The current case study approaches are presently just too expensive and subjective to contemplate system-wide. Nevertheless, an obvious gap exists in current indicator-based approaches to capture and measure impacts of research from Indigenous perspective. There is an urgent need for working with (rather than on) Indigenous people in order to “overcome the continuing embeddedness of colonising practices in reductionist indicator-based approaches” [
55] and develop effective local Indigenous scale indicators to capture and measure intangible impacts and benefits of Indigenous research.
Gaining consensus about the quality and nature of evidence can be divisive [
24]. Methodologically, less value is placed on Indigenous and local knowledge and narratives of change. However, inclusions of these are imperative in establishing meaningful and relevant indicators to measure impact; they are particularly important in assigning meaning to experiences of inequality, implementation and change processes and understanding the context in which such indicators should be developed. The importance of context is exemplified in the research impact model developed by Kuruvilla et al. [
23] which supports a multi-level approach to assessing impact including research-related, policy, services: health and intersectoral and societal impacts. In each of these areas, descriptive elements are identified for modification or tailored to the context in which the measurement of impact will occur. While researchers are preparing to push the boundaries of research to contemplate a more rounded approach to assessing benefit, internationally, countries such as the UK are looking to economic evaluations. However, Australia is lagging behind in this regard; for example, the NHMRC does not request that people look at return for investment or value for money in their applications for funding. It is however evident in Cooperative Research Centre schemes where applications explicitly call for grant applicants to model and monitor the longer-term economic benefits of their research. This is another important area to be explored in planning the assessment of impact and benefits.
Silburn et al. [
39] cite some of the many challenges faced in measuring the impacts of Indigenous health research. These are precisely the barriers to doing policy relevant impact/benefit measures globally. Challenges include: 1) attribution, such as the multitude of factors influencing health and health outcomes and the difficulties in attributing changes in outcomes to changes from factors; 2) identifying the variety of ways health research can influence outcomes; 3) the time lag between the outputs and potential outcomes; 4) the meaning ascribed to change may not be obvious (for example, more investment in a particular condition might mean that it has become more prevalent or more severe, or that there are more available diagnostic tests; 5) that links from research to impact are generally not linear; 6) difficulties in attributing economic value to aspects of health; 7) lack of records documenting how research is taken up by end-users; 8) some measures of change such as knowledge, develop incrementally and often result from multiple research projects across time; and 9) developing frameworks that account for the priorities of various different stakeholders (for instance, while measuring economic impact might be important for government; the same premium on one indicator is not likely to be valued by community members.