A systematic review of studies evaluating Australian indigenous community development projects: the extent of community participation, their methodological quality and their outcomes

The peer-reviewed and grey literature were searched to identify studies evaluating Indigenous community development projects in Australia, published between 1990 and 2015. Twenty-five years of community development projects was judged to be sufficient to provide an overview of the most recent projects. Figure 1 summarises the databases searched, the search terms used, the eligibility criteria and the classification process based on the PRISMA flow diagram [27].

Ten peer-reviewed databases were searched: Health and Society, Aboriginal and Torres Strait Islander Health Bibliography (ATSIHealth), AIATSIS, APAIS-AIATSIS, FAMILY-ATSIS, ProQuest, Scopus, CINAHL, PsychINFO, and Medline/Pubmed. Two grey literature databases were searched: HealthInfonet and Closing the Gap Clearinghouse. The electronic database search identified 3623 publications and 411 duplicates were removed. Reference lists of the identified publications were scanned which identified an additional 20 publications. Another 11 publications were received from researchers in the field. The resulting 3243 publications were organised in Reference Manager Endnote [28].

The titles and abstracts of the identified 3243 publications were read to determine their eligibility for inclusion against three criteria: 1) relevance to a community development project, including: projects focusing on community ownership, empowerment, local leadership and decision making, adopting a long-term strategy, having a focus on sustainability or having a bottom up approach (i.e. starting from the community) [29, 30]; 2) published between 1990 and 2015; and 3) a primary focus on Indigenous communities in Australia. A total of 231 publications met all three criteria. The full text versions of these 231 publications were sought for detailed review, of which 112 were available and relevant to this review.

One hundred and twelve publications were classified into four categories derived from previous research reviewing Indigenous health initiatives [4, 10, 31], defined as follows. Measurement research: the development, testing or evaluation of measurement tools. Evaluation research: an evaluation of an Indigenous community development project or policy. Review: including summaries, critical or systematic reviews and/or meta-analysis; and Discussion paper: general discussion of Indigenous community development. Thirty one publications identified as studies evaluating community development projects in Indigenous Australian communities were critically appraised.

Replicating previous analyses [32, 33], the extent of community participation was assessed using Pretty’s participation typology, which describes seven levels of community participation ranging from no participation to self-mobilisation (i.e. completely top-down to completely bottom-up) [32, 34]. Given community participation can vary during the lifetime  of a project, the extent of participation was assessed separately for four phases of project development: diagnosis (identifying a community’s priorities); development (of appropriate strategies to address the priorities); implementation (of the strategies); and evaluation (of the effectiveness of the project) [32, 35, 36]. The level of community participation in the 31 studies were assigned a score between 1 and 7 for each phase of project development. Detailed descriptions of different levels of community participation in relation to scores 1 to 7, and with respect to all four phases, are provided in Table 1 and are summarised as follows: no participation (score 1); passive participation (score 2 – the community was only informed about the project); participation by information (score 3 – information was collected from the community without their participation and without providing feedback); participation by consultation (score 4 – information was collected from the community, feedback was given and further inclusion of community was sought); functional participation (score 5 – community collaboration, but on outsiders’ terms); interactive participation (score 6 – collaboration on mutually defined terms); and self-mobilisation (score 7 – outsider’s work in community on community’s terms). This scoring was independently conducted by two of the authors (MS and AW), which resulted in agreement for 22 studies (71 %). The nine studies on which authors disagreed were blindly reviewed by a third author (ASt). For four studies, the score allocated by the third reviewer was the same as the score allocated by one of the first two reviewers and so that score was used, increasing the agreement rate to 84 % for 26 of the 31 studies. For the five studies where there was no agreement, the scores allocated by the first author (MS) were used.

The outcomes of the studies evaluating Indigenous community development projects are summarised.

Table 2 summarises the level of community participation across the four phases of project development for each study. Table 3 summarises the number of studies relevant to each of the seven levels of community participation, separately for the four phases of project development. The highest levels of participation (level 5 to level 7), were found in the Diagnosis phase for ten studies (32 %) [39, 43, 44, 48‐54], in the Development phase for 13 studies (42 %) [39‐44, 46, 49‐52, 54, 55], in the Implementation phase for 17 studies (55 %) [39‐44, 46, 48‐54, 56‐58] and in the Evaluation phase for 7 studies (22 %) [39, 40, 43, 44, 50, 53, 59]. Four studies (13 %) had at least level 5 participation in all phases of the project [39, 43, 44, 50]. The participation of the community was described with insufficient detail to be assessed (unknown category) for ten studies in the Diagnosis phase (32 %) [42, 45, 57, 58, 60‐65], seven in the Development phase (23 %) [47, 48, 57, 60, 61, 64, 66], four (12.9 %) in the Implementation phase [47, 61, 63, 66] and two (7 %) in the Evaluation phase [46, 67].

Twenty-one studies (67 %) used qualitative methods only [39‐41, 48, 49, 51, 52, 55‐57, 59‐69], two (7 %) used quantitative methods only [46, 47], and eight (26 %) used mixed methods [42‐45, 53, 54, 58, 59]. Qualitative data were collected using semi-structured interviews in 24 studies [39, 40, 42, 43, 45, 49‐53, 55‐58, 60‐69], document analysis (n = 15 [42, 49, 51, 52, 55‐57, 60‐66, 68]), focus groups (n = 9 [39, 40, 43, 45, 48‐50, 53, 58]), participant observation (n = 6 [42, 43, 53, 54, 58, 69]) and photovoice [70] (n = 2 [50, 53]). Quantitative data collection methods included surveys in three studies [45, 48, 58], hospital/clinical records (n = 4 [43, 44, 47, 53]), school records (n = 2 [42, 58]), police records (n = 1 [42]), store records (n = 1 [43]) and ABS census data (n = 1 [46]).

All 29 studies with a qualitative component (including mixed methods studies) provided some description of the evaluation methods used (Table 4). Twelve studies (41 %) gave detailed descriptions of the data collection process, including participant recruitment, focus group procedures and a clear description of which data were recorded [39, 41, 49‐51, 53, 58, 60‐63, 67]. Four of these twelve studies (14 %) provided the interview questions [51, 58, 60, 67] and one study (4 %) described in detail how the data collection methods were tailored to ensure their cultural appropriateness [49]. The data analysis methods were described in detail in seven studies (24 %) [39, 42, 50, 54, 58, 67, 69]. The potential for researcher bias was described in seven studies (24 %) [39, 44, 45, 49, 53, 58, 69]. Three studies (10 %) did not discuss the implications of their findings [52, 59, 65].

The summary ratings for all ten studies with a quantitative component were classified as weak (Table 5). The likely extent of selection bias was unclear for six studies (60 %) because description of the participant and community selection procedures was absent or insufficiently detailed [42‐44, 46, 47, 59]. Five studies (50 %) used a cohort design without a control group [43‐45, 53, 54], one study (10 %) used a time series design [47] and the evaluation design of the remaining four studies (40 %) was unclear [42, 46, 58, 59]. No study adequately controlled for confounding variables. None of the studies used blinding procedures. Two studies (20 %) used validated outcome measures [45, 58]. No study discussed the validity or reliability of their outcome measures.

One study (10 %) described withdrawals and drop outs at the community level [46]. One study (10 %) described withdrawals and drop outs at the participant level [53]. Drop outs at the participant level were not applicable for the six studies (60 %) that used either routinely collected data or a one-off survey [42‐44, 47, 54, 58]. Two studies (20 %) did not report drop outs [45, 59]. No study described the fidelity of the project. Three studies (30 %) reported on the exposure of participants to the project [45, 54, 58]. Inferential statistical analyses were reported by four studies (40 %) [42, 47, 53, 58].

A summary of the aims and key outcomes for each study is provided in Table 6. All studies using qualitative methods concluded that community members reported positive project impacts for their community. Two studies (7 %) reported quantitative outcomes that were statistically significant: a reduction in injuries [47] and a reduction in cannabis use among females aged 13–36 and males aged over 16 years [42].

Community participation was assessed as moderate in most of the studies evaluating Indigenous community development projects (87 %). The wide variation in community participation between projects and project phases, and within project phases, in these Australian Indigenous studies is reflected in the international literature [32, 36]. For half of the studies included in this review, the intent for community participation was clearly described, but the actual level of participation was not reported for at least one of the phases of project development. Documenting the community participation strategies and processes used, including details about how the community was engaged and who in the community participated, would allow the more successful community participation strategies to be identified and replicated in subsequent projects [36].

Although the unique characteristics of each community will lead to variation in their capacity to participate in each phase of a project [36, 71], the extent and nature of community participation can be optimised by careful planning and the utilisation of appropriate frameworks to guide the development, implementation and evaluation of community-based projects. An approach like participatory action research provides practical guidelines to achieve this [72, 73].

In line with previous research [4‐12], the methodological quality of the studies identified in this review are poor, or they are difficult to assess because their methods are inadequately described. It is acknowledged that issues specific to Indigenous community-based research can impact on the research quality, including time needed to engage with the community, difficulties with recruiting enough participants, high staff turn-over at service providers and culturally-specific delays (e.g. ceremonies or celebrations) [14]. Careful and flexible planning is therefore needed in community development projects to address these issues to reduce their impact on the quality of the research. The complex interventions framework, for example, provides one mechanism to carefully plan projects to maintain methodological rigour [74, 75]. The methodological quality of qualitative studies could also be improved by using appropriate analysis methods, multiple coders, and describing the extent of potential bias attributable to the researcher [37].

The methodological rigour of both the qualitative and the quantitative studies could further be improved by using measures with demonstrated reliability and validity: only two studies identified by this review reported that they had used such measures [45, 58]. Using reliable and valid measures increases confidence in the accuracy of the study outcomes [76]. Such measures should be validated specifically for the Indigenous population, because of their holistic concept of health and wellbeing [77]. Existing studies show that it is possible to develop reliable and valid measures that are culturally appropriate and acceptable to Indigenous Australians [78‐83], but the lack of measurement studies specifically related to community projects identified in this review (one study, see Fig. 1) clearly indicates that more of this measurement research is urgently needed [80].

Only three studies (10 %) reported on intervention integrity, which includes the level of exposure to the project, and the consistency and frequency with which project components were delivered in practice. Studies evaluating community development projects would be improved by routinely including process measures, to allow an examination of the extent to which outcomes are a consequence of the project components, as opposed to reflecting the extent to which the project components were implemented [74].

Eight studies (26 %) evaluated a community development project using a mixed methods design. Increasing the use of mixed methods is likely to optimally improve the effectiveness of future community-based evaluations because they provide a greater range of relevant data [11, 84]: quantitative analysis can provide rigorous methods to evaluate the effectiveness and costs of projects, while qualitative data can capture community members’ experiences [85] and help identify the project elements that are most acceptable to community members [11].

The critical appraisal also identified a lack of detailed reporting of the methodologies used, especially in relation to the qualitative evaluations. Only 41 % of the qualitative studies reported on their data collection process, for example, and only 24 % reported the data analysis methods that were used. Future Indigenous community development evaluations would benefit from more detailed reporting using established guidelines, such as the COREQ criteria for qualitative research [86] or the guidelines recommended by the Equator Network [87]. In addition to improving reporting standards, using these guidelines in the development, implementation and evaluation phases of community development projects would most likely improve the quality of the interventions and their evaluation [37, 38].

There is currently insufficient evidence about the impact of community development projects on health and wellbeing outcomes for Indigenous Australians. Although all reviewed studies reported positive outcomes for the communities, they are not methodologically rigorous enough to support clear conclusions about their cost-effectiveness, and no studies have undertaken an economic analysis to weigh the benefits of community development against its costs. This finding is highly consistent with the conclusions of similar reviews of international Indigenous community development studies, where generally positive outcomes are difficult to interpret because of the relatively poor quality of their evaluation designs and reporting [9, 19, 88]. Published results of community-based evaluations with greater methodological quality are required to provide evidence of cost-effective community development projects [9, 74]. Ideally, future studies would use rigorous evaluation designs, reliable, valid and culturally appropriate measures, economic analysis and a complex intervention framework to balance standardisation and tailoring.

To ensure that qualitative and quantitative study components were assessed against appropriate criteria the Dictionary for Effective Public Health Practice Project Quality Assessment tool [38] was used to assess the methodological quality of quantitative components and an adaptation of the qualitative study appraisal tool, developed by Long and Godfrey [37] was used for the qualitative study components. The methodological quality of the studies and extent of community participation may have been misclassified, however the high level of agreement between blinded coders suggest not. Of the 231 full-text articles sought for detailed review, 40 (17 %) were excluded because the full text version of these papers were unable to be accessed. Excluding these 40 papers is unlikely to have compromised the comprehensiveness of this review for three reasons: 1) they only represent 17 % of the full-text articles; 2) the majority were older studies or reports that were not publically available; and 3) the references lists of identified publications were hand searched and researchers in the field were consulted to identify publications not found by the electronic database search.