Background
The nature, quality and value of research evidence in positively impacting and raising Indigenous health status can justifiably be questioned when significant gaps in health outcomes between Indigenous people and benchmark populations persist globally [
1,
2]. Questioning the way researchers do business is also being challenged by Indigenous people who continue to voice concerns about being over-researched without corresponding improvements in their health [
3]. Relatively unchanged long-term health disparities for Indigenous populations strongly suggest that despite the plethora of health research, corresponding research impacts or benefits for Indigenous people are, at best, modest. As researchers, we have a moral obligation to be responsive to need and ensure that we deliver high impact research evidence from which societal benefits can flow. Research end-users such as policy-makers and practitioners need to know how much confidence they can place in research evidence to make smarter informed healthcare decisions [
4]. In tight contemporaneous fiscal environments, ensuring quality evidence also makes sense from a cost perspective. In this paper, we focus on the quality of evidence reported in systematic reviews of Australian Aboriginal and Torres Strait Islander (hereafter respectfully Indigenous Australians) health research as a necessary prerequisite to effecting societal impacts and benefits. A systematic review of reviews on Indigenous Australian health research was conducted to assess the quality of research evidence and the extent to which impact and benefits were reported.
Following Bainbridge et al. [
3] and for the purpose of this review of reviews we consider “research impact” and “research benefit” as discrete terms that are interrelated and interdependent. Research impact is defined as any area of influence flowing from the research endeavour including those that flow from research processes; these can be both positive and negative. Research benefit flows from areas of impact. In Indigenous health research, benefit is broadly defined as any elements of research that are advantageous or good; such as strengthening capacities, opening opportunities, or improving health outcomes that progress the interests that are valued by Indigenous people [
5]. Benefit can be intended or unintended, inside or outside the immediate research environment, direct or indirect, tangible or intangible, immediate, short-term or longer-term; but benefit must be positively oriented and represented as elements of value derived from research [
3]. Herein we use the term “impact” as the first step for attaining benefits, but look at its positively oriented nature, as in research benefits.
Silburn et al. [
6] noted that the likelihood of favourable research impact and thus benefits is related to the ethical implementation of Indigenous health research processes. Maximising research impact is more probable when: 1) Indigenous people are direct and active participants; 2) issues of relevance to Indigenous people are identified and prioritised; 3) Indigenous peoples’ knowledges and perspectives are incorporated in processes and findings; 4) reporting of findings are meaningful; and 5) all potential end-users are engaged from the outset. Yet, it is unclear whether research is implemented ethically or whether the abundance of research conducted, purportedly to improve Indigenous health, is justified and produces quality evidence that benefits Indigenous people in ways that are meaningful and valued by them [
3].
Aim and hypotheses
This systematic review of reviews extends the author team’s prior work to elucidate the reporting of Indigenous research impact [
3]. It synthesises evidence from multiple reviews in a variety of settings to provide information concerning the extent to which impact is examined across different health areas, programs and interventions. Based on the domains of the Research for Impact tool [
4] developed for Australia’s national Indigenous health research organisation, the Lowitja Institute, a set of three a priori hypotheses were developed to test the extent to which published reviews make transparent the impact of research studies on Indigenous health that are included in their reviews. The Tool provides a step-by-step research logic framework designed to increase the value of research in Indigenous Australian health. It is based on the assumption that the implementation of the six objectives in the framework will lead to demonstrable short, medium and longer term impact. The performance or impact of research projects identified in this review of reviews were evaluated against these six objectives: 1) promoting Indigenous leadership and participation; 2) ensuring rigorous and transparent research priority setting; 3) ensuring the study type and design are appropriate to the research question(s); 4) ensuring respectful and rigorous implementation; 5) facilitating knowledge translation into policy and practice; and 6) determining the impact of the research versus its costs. Consistent with these objectives, the hypotheses developed for this systematic review of reviews are:
1.
Reviews address Indigenous Australian health priority needs;
2.
Reviews adopt best practice guidelines on research conduct and reporting in respect of methodological transparency and rigour, as well as acceptability and appropriateness of research implementation to Indigenous Australian peoples; and
3.
Reviews explicitly report the incremental impacts of their included studies and translation of research.
The overarching intent of the systematic review of reviews is to influence the current research reporting and accountability frameworks and expectations towards greater focus on improving the quality of evidence and reporting research impact. This study complements other review of reviews of Indigenous health [
7,
8] by focussing explicitly on the impact of research. It also extends the search scope from peer-reviewed to policy initiated reviews (resource sheets), considers reviews’ output over time, and provides characteristics of included studies and institutional affiliations. The review therefore provides researchers, policy-makers, health practitioners and other decision-makers with directions towards greater focus on reporting and demonstrating research impact–the knowledge they need to implement evidence-informed health and wellbeing improvements.
Methods
In recent years, health research has been assessed through reviews of the literature. Reviews help to make evidence more accessible by bringing research together in one place [
9]. We worked from the premise that judgements about the quality of evidence could best be assessed through systematic review methods. We followed the established PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) guidelines [
10] for designing and preparing the manuscript. An a priori protocol defined the eligibility criteria, information sources and search strategy, outlined the data collection process, synthesis of results and quality appraisal strategies.
Eligibility criteria
Studies included: (1) any type of review, for example literature review or meta-analysis or meta-synthesis, meta-ethnography or narrative synthesis, systematic search or bibliometric analysis; (2) focused on Indigenous Australians’ health and wellbeing; (3) published between 1995 and 2014 (inclusive); (4) in the English language; and (5) available as full-text manuscripts.
Indigenous health and wellbeing was defined broadly as “the social, emotional and cultural wellbeing of the whole community in which each individual is able to achieve their full potential as a human being, thereby bringing about the total wellbeing of their community, but not just the physical wellbeing of an individual” [
11]. This broad definition meant that studies were included if they addressed a health-related process such as mentoring, health promotion, or a determinant of health such as education or housing initiatives, provided they met all other criteria. Reviews that were international in scope and crossed over Indigenous and tribal populations across various countries (
n = 8, 11%) and reviews that considered Indigenous people as a part of a rural or a socially disadvantaged population (
n = 3, 4%) were also included, if they reported results separately for Indigenous Australians.
As a quality indicator, this systematic review of reviews was limited to peer-reviewed literature. However, it was acknowledged that grey literature “may contain rich texts and their inclusion may avoid publication bias” [
12]. With this caveat in mind, one pertinent sub-section of the peer-reviewed grey literature, known as Closing the Gap Clearinghouse, was included. Resource sheets and issues papers in the Clearinghouse contained evidence on narrowly defined topics commissioned by the Australian Institute of Health and Welfare and the Australian Institute of Family Studies (
https://aifs.gov.au/). The aim of the Closing the Gap policy was to achieve equality in health status and life expectancy between Indigenous and non-Indigenous Australians. Resource sheets were subjected to peer review processes and considered to be of high relevance to Indigenous policies and programs.
An accredited librarian working at a leading Australian university designed and carried out a comprehensive search strategy of: (1) 14 electronic databases: Informit, Indigenous Australia, Indigenous Studies Bibliography: AIATSIS, APAIS-ATSIS, FAMILY-ATSIS, Informit Indigenous Collection, ATSIHealth, Campbell Library, EBM Reviews/Cochrane DSR/ACP Journal club/DARE, PsycINFO & PsychARTICLES, Medline, Ovid, CINAHL, Sociological Abstracts, Australian Indigenous HealthInfoNet; and (2) the Closing the Gap Clearinghouse website with a catalogued repository of evidence based resource sheets.
Search
The following terms were searched in either title, abstract or MESH heading: systematic AND Australia AND review AND health AND (Aborigin* OR Indigenous OR “Torres Strait”). MESH terms comprised meta-analysis, oceanic ancestry group, Indigenous health services, Australia; subject terms included meta-analysis, Indigenous peoples OR Aborigines and health. The database search was performed in December 2014. The search was conducted separately for each database and use database specific search strings.
Discussions with experts established that systematic reviews in the field emerged as a common research type only in recent years. Therefore, selected period from 2004 to 2014 (inclusive) and from 1995 to 2014 for Medline were considered appropriate to provide comprehensive coverage of relevant reviews. The search of the Closing the Gap Clearinghouse website was conducted in May 2015.
Data collection and analysis
Information was extracted from each included review for: first author, year, title, institutional affiliation and country of the first author; type of review; period searched (P), number of studies (K) and type of studies analysed; health area; target group: gender, age; reasons for the review; major findings (review authors’ conclusions); assessed research impact; review design; quality of included studies; compliance with Indigenous health research ethics.
Attention was devoted to understanding how researchers reported the impacts of their research for Indigenous people and how such impact was framed. The full texts of the reviews were searched for keywords such as impact, benefit, value, effect, consequence, assess* and translation.
Health areas were categorised using the socioeconomic determinants of health framework by Turrell et al. [
13] and broadly divided into seven themes (areas): biological, psychosocial, health behaviour, healthcare system, health research, health inequality, setting and contexts. The authors of this study categorised reviews into as many health related areas as were relevant. Therefore, if a review fitted into more than one area, all were recorded.
An Excel (Microsoft Corporation) Datasheet 2013 was utilised to record the data items. One review author (IK) extracted the data from included studies and the second author (KT) cross-checked the extracted data. Two authors (IK, KT) independently assessed the included reviews. The coefficient of agreement was
k = 85%,
z = 16.15,
p < 0.001, indicating a substantial level of agreement on extracted and recorded data items [
14]. Discrepancies were resolved by discussion.
Quality appraisal
Quality appraisal comprised a six-step analysis. Step 1: type of review was determined and considered as a quality measure [
15]. Step 2: the authors ascertained the use of reporting guidelines, for example Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) or any other relevant guidelines. Step 3: the reviews identified in the previous step were assessed with the AMSTAR tool [
16], a validated measurement tool for assessing the methodological quality of reviews. The tool contains 11 questions (domains) (see Additional file
1). Each included review was assessed against all the domains with assessments labelled as ●, ◐ or ○. Reviews that fully addressed the domain were labelled ●; reviews that addressed the domain to some extent were assigned ◐; and reviews that either did not provide enough information for quality analysis, or the answer could not be ascertained, were assigned ○. Step 4: the authors investigated whether reviews assessed quality of the primary studies they had included. Step 5: the authors examined methodological rigour of the reviews through utilisation of quality appraisal tools informed by ethical standards and guidelines for conducting research with Indigenous people for the purpose of identifying the existence and utility of such tools. Step 6: lastly and importantly, the authors examined whether reviews documented Indigenous leadership and/or participation in research. Full texts of the reviews were examined for acknowledgment of Indigenous origin among authors and cross-checked by two authors (RB, JM) of this review.
Synthesis of results
Due to the heterogeneity of health areas, interventions and outcomes, a quantitative summary measure of the results was not planned. Instead, descriptive analysis was used to synthesise results which were then presented according to the three hypotheses outlined in the introduction.
Discussion
This systematic review of reviews on Indigenous Australian health research analyses the quality of evidence and reporting of research impact in existing health reviews. We argue that if review authors explicitly address each of our three hypotheses, then the impact of research for Indigenous health would be explicated. While reviews are gaining prominence and recognition in Indigenous health research as an important methodological approach to dealing with ever growing amounts of research output, and are increasing in number, breadth and complexity, we found that currently, there is little reporting of the impact of health research for Indigenous people and that few met the criteria expected in the reporting of quality evidence. These findings raise questions about the relevance of these reviews for Indigenous people, their impact on policy and practice and how reviews have been commissioned, reported and evaluated.
Although the priorities for Indigenous health are arguable and vary from region to region, on the whole the findings of this systematic review of reviews partially support the first hypothesis: that reviews of Indigenous health address Indigenous health priority needs. This systematic review of reviews considers health broadly as a whole-of-life view associated with wellbeing of individuals and their communities and, to our knowledge, is among the first reviews that focuses not only on health from the Western perspective, but also on the important social determinants of health. To date, there is still a continuing predominance of reviews focussed on biological or clinical models of health which is promising but not sufficient for addressing Indigenous health. Social and emotional well-being encompasses all aspects of the social and emotional context of the person and their family, the historical and economic factors, including racism, oppression, trauma, grief, loss in its many forms, and the sequelae of the Stolen Generations, and it therefore allows for a more holistic framing of health that needs to be recognised in Indigenous research and its impact [
47]. Further reviews of aspects of Indigenous mental health and social and emotional wellbeing are needed. Likewise cultural knowledge and context cannot be fully understood without Indigenous peoples’ involvement in research, participation and leadership which clearly has not been acknowledged and documented sufficiently in the analysed reviews.
Our second hypothesis, that reviews should adopt best practice guidelines on research conduct and reporting in respect of methodological transparency and rigour, as well as acceptability and appropriateness of research implementation to Indigenous peoples, was also rejected. Reviews have progressively become a mainstream methodology in Indigenous health research. Nevertheless, the current analysis reveals a clear misinterpretation as to what is understood by the term “review” and, in particular, “systematic review”. Nearly two thirds (63%) of the systematic reviews, which by definition claim to have a rigorous approach to searching, evaluating and reporting, have not reported nor followed any guidelines. Undertaking a systematic review requires a robust methodology. Authors who endeavour to undertake a systematic review are expected to meet detailed and rigorous requirements and follow guidelines or standards on the conduct and reporting. These include the development of an a priori protocol, which is submitted for peer review prior to the commencement of the review; undertaking rigorous searches which align with the inclusion and exclusion criteria specified in the protocol; and strict alignment with guidelines for the preparation of manuscripts for publication [
52].
What was reported and how it was reported also varies considerably. The majority of the reviews were based on qualitative analyses of descriptive and intervention studies. Cooper et al. [
53] observed that the traditional review process typically “lacks analytical precision because of biases associated with a reviewer’s idiosyncratic perspective, failure to assess the size of the effects reported by studies reviewed, and imprecise combination of the volume of evidence available across the studies reviewed”. According to Cooper et al. [
53], a synthesised approach such as meta-analysis can offer a method that avoids some of the problems of the traditional literary summary. Synthesised approaches were underutilised in Indigenous health reviews. Having said that, reviews concerning Indigenous health are complex in nature with prevalent attribution issues; it is hard to establish the impact of a particular intervention if multiple simultaneous interventions are being delivered. Furthermore, Indigenous health is an area where randomised control trial interventions, with outcomes presented as quantified measures that can be subjected to meta-analysis, are not always ethically appropriate. All of these factors contributed to relatively weak evidence base in Indigenous health research.
The rigour of the research from the perspective of Indigenous people directly relates to scarcity of Indigenous leadership and/or participation. Concerns exist that the reporting of reviews has often been culturally inadequate. A few attempts have been made (four reviews out of 76) to assess the ethical appropriateness and rigour of the research from the perspective of Indigenous people, however, no systematic way of appraisal was found. Authors either developed and utilised a context scale tailored to a specific health area [
37,
38,
44] or applied a generic tool, not specifically designed to align with an Indigenous research context [
45]. This may reflect either, or both, a lack of awareness of reporting guidelines or tools informed by Indigenous peoples’ perspectives on ethically appropriate and rigorous research processes and outputs, or a perception that guidelines that first emerged from clinical medicine are not applicable in the social sciences where most Indigenous health research is being conducted. As pointed out [
7], absence of such tools might lead to a waste of resources on reviews with limited relevance for Indigenous people. The marriage of health research to needs can only occur with consultation, engagement and the trust of Indigenous communities. We recognise the need for a cultural diversity tool specifically designed for conducting systematic reviews in the context of Indigenous health.
Finally, the third hypothesis, that reviews explicitly report the incremental impacts of included studies and research translation, was also rejected. Even though review authors might state an aim to influence policy or practice, often it was mostly unclear who or what they were attempting to influence, or the impact of their research. Dialogue at the nexus between research, policy, and practice is crucial if systematic reviews are to make a significant impact on practice. Instead, reviews need to be framed in such a way that they feed into the development and refinement of policy and practice guidelines which “interpret results to make knowledge of intervention effects available to a wide audience, including readers who are unfamiliar with the methodological and technical aspects of both primary research and research synthesis” [
52,
54]. The identified lack of reporting research impact makes the decision by the Lowitja Institute to develop such a tool that would support planning and reporting impact of research over time responsive to the need and very timely [
4].
Limitations
While this paper provides a systematic review of the state of the Indigenous health research, because it is a systematic review of reviews, it does not include potentially useful primary studies that have not been included by the identified existing reviews. Reviews published prior to 1995 nor those published since 2015 have not been included. The search strategy could have been improved by adding a more sensitive search filter for Indigenous health related terms. It is possible that some reviews are excluded as a result; however, it is unlikely that the overall findings would be significantly different. We drew on the authors’ knowledge of Indigenous health research to identify additional studies. The authorship included one Aboriginal author (RB) and one Torres Strait Islander author (FWL). We acknowledge the limitations of our approach in addressing the impact of research though analysis of reviews rather than the original studies. We cannot rule out the possibility that original studies reported the impact of their research. However, if such had been reported in original studies, we assumed a high probability of reporting the impact by reviews as well. We encourage further research and analysis of original studies to address this limitation.
We also acknowledge the lack of a benchmark or a threshold figure for testing the research hypotheses. This was mainly due to the absence of any formal numeric criteria. Authors used their professional assessment to gauge whether hypotheses could be rejected or not. Lastly, the methodological quality of included reviews is assessed using the AMSTAR tool designed for studies in any population, not specifically Indigenous due to the nonexistence of Indigenous specific tool at the time of publication.
Acknowledgements
We would like to acknowledge Lowitja Institute for providing funding to support this review and James Cook University for in-kind contribution.