Introduction

Community-based Rehabilitation (CBR) is an umbrella-term for strategies “within general community development for rehabilitation, equalization of opportunities, and social inclusion of all people with disabilities” that aim to address their wider needs in their communities. CBR is implemented through the combined efforts of people with disability themselves, their families and communities, and the relevant service sectors[1]. CBR is implemented in over 100 countries, evolving from its initial focus on limitations and barriers experienced in low-and middle-income countries to also be relevant for higher-income countries[2,3]. However, CBR coverage is usually very low regarding the proportion of people with disability receiving support, CBR is seldom integrated into health or social security systems, and is instead usually financed and provided by non-governmental organizations (NGOs)[4]. Acknowledging the importance of CBR in tackling stigma, discrimination, barriers to equal participation, and lack of appropriate services faced by people with disability[5], the World Health Organization (WHO) set strengthening CBR, particularly through fostering the improvement of CBR monitoring and evaluation, as one objective of the recently endorsed Global Disability Action Plan[6].

Sound and systematic CBR monitoring and evaluation is a significant challenge faced by the CBR sector in promoting and advocating for its broader implementation. While anecdotal evidence exists on the success of CBR, internationally comparable results are still rare, and reliable and comparable data needed to monitor and evaluate CBR scarce[4,7,8]. Although the existing qualitative work delivers essential in-depth understanding of the changes CBR initiates[9], the lack of standardized measures limits the generated evidence and the comparability across settings[7,10]. A recently published systematic review reporting evidence on the effectiveness of CBR in low- and middle-income countries pointed out promising results in terms of clinical outcomes, functioning and quality of life, but could not deliver clear evidence due to the heterogeneity of interventions and quality of included studies[4]. A second review examining the methods used to collect data on CBR programs corroborates the lack of standardisation. This particular review calls for the development of a data collection method which takes the complexity and heterogeneity of CBR into consideration while keeping a high level of standardisation[7].

Indeed, several attempts have been taken towards developing standardized data collection methods for CBR, by attempting to identify reoccurring CBR domains, to suggest evaluation frameworks, and to develop specific indicators. In 1995, a joint WHO workshop looked to develop outcome indicators with the goal of providing qualitative information about the effectiveness of CBR activities, with a special effort to create indicators beyond the health component of CBR[11]. One of the first attempts to introduce the use of classification models to evaluate CBR was suggested in 2000, which used four dimensions with a defined scoring system to categorize programs[12]. A few years later, Wirz and Thomas noted that many studies have attempted to compile sets of indicators to judge the effectiveness of CBR. Based on ten included studies, they identified six activity domains and derived indicators in line with these activities[13]. One year later, a workshop developed a template that comprised of a number of guiding questions within three domains, which were then later developed into a set of evaluation indicators[14]. In 2010, the release of the WHO’s Community-Based Rehabilitation Guidelines (hereafter CBR Guidelines) served to synthesize global perspectives on CBR, and have since become accepted internationally as a conceptual framework for CBR[15].With these guidelines, the WHO recognized that no single model of CBR is appropriate for the whole world and suggested the pre-existing CBR Matrix (Fig 1) as a common framework to reflect the comprehensive multi-sectoral strategy that is CBR. In 2012 the CBR Guidelines were used during a WHO technical meeting[16] as a guide to develop a set of indicators, focusing mainly on access to CBR services, and being in this sense restricted to a single perspective. Furthermore, consensus was not reached regarding these indicators, and they were therefore not promoted for use. A CBR Monitoring Manual and Menu, published in 2015, outlines possible methods, encourages the setup of easy and routine monitoring and provides information that can be used as building blocks for indicators. However, standardized indicators are not presented[17].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. The CBR Matrix.

https://doi.org/10.1371/journal.pone.0178418.g001

Despite these efforts towards standardization for monitoring CBR, a standardized multi-sectoral and internationally comparable set of CBR indicators suitable to monitor the changes that different CBR strategies initiate in the lives of people with disability, is lacking[4]. Previous CBR indicators, however, tend to describe practices and stakeholder perceptions, rather than asses the changes brought by these practices[13]. Furthermore, they tend to target specific CBR activities or are tailored for a specific region[14,18,19]. Indicators suitable to monitor CBR across communities and countries need to be generic in that they can be used to monitor CBR independent of the specific objectives and activities of individual CBR programs. Also absent from previous CBR indicator initiatives is the possibility of comparing people with disability and those without disability living in the same community though the use of the same indicators. Using people without disability living in the same community as a reference group is necessary in order to disclose inequalities and changes in inequalities when CBR is in place. This is of utmost importance in order to be in line with the United Nations Convention on the Rights of Persons with Disabilities (CRPD), which states that people with disability should have equal rights as everyone else[20]. Finally, since the internationally accepted CBR Guidelines reflect the comprehensive multi-sectoral CBR strategy, indicators based on these guidelines would be the most updated way of monitoring CBR following the five components presented in the CBR Matrix: health, education, livelihood, social and empowerment[15].

Accounting for this need, WHO initiated a project to develop CBR indicators encompassing health, education, livelihood, social life, and empowerment indicators in accordance with the CBR Matrix. In the practice of CBR, input and output indicators do not provide an indication of the extent to which a program is achieving its goals or its effectiveness[21], while impacts are long-term effects that are beyond attribution to a CBR intervention as they may reflect broader societal change. Therefore, the CBR indicators were intended to be outcome indicators, as outcomes are the observable short- and intermediate-term changes in a specific group of persons in the CBR area that have been influenced by the outputs. These outcome indicators will serve to standardize monitoring of CBR across areas and countries. The adopted working definition of monitoring was: a descriptive process that provides information on the state of a programme or project at a given time, relative to its respective targets and outcomes[21]. To ensure that different perspectives in terms of CBR expertise were taken into account, and to thereby increase the uptake of the proposed indicators, the entire development was a consensus process in collaboration with the International Disability and Development Consortium (IDDC), stakeholders in the greater CBR community, as well as a team of external researchers.

The aim of this paper is to describe in detail all phases of the development process of the set of global WHO CBR outcome indicators, which were launched by WHO in December 2015. This WHO document, “Community-based rehabilitation indicators manual” (available at http://www.who.int/disabilities/cbr/cbr_indicators_manual/en/[22]), presents the indicators and the recommended data collection and reporting strategy. While the manual is meant to be a “how-to” guide for using the indicators, this manuscript presents the details of the development process for transparency.

Methods

CBR Guidelines were used as the underlying framework for developing the WHO CBR Indicators because they are internationally accepted as conceptual framework for CBR, were developed together by WHO, IDDC, the United Nations Educational, Scientific and Cultural Organization (UNESCO) and the International Labour Organization (ILO) and are based on a worldwide participatory approach with the involvement of all relevant stakeholders[15]. The CBR Guidelines synthesize global perspectives on CBR and offer recommendations on strengthening the capacity of mainstream and specific services to include people with disability in accessing the benefits of the health, education, livelihood and social sectors and enhance empowerment. For each of these five components of the CBR Matrix (Fig 1), there are five corresponding elements for which the CBR Guidelines present a set of “desirable outcomes” outlining the goals of CBR. Additionally, general overarching desirable outcomes are presented for each component. These desirable outcomes were used as a starting framework for the WHO CBR Indicators. The development process comprised four phases with specific objectives as follows.

Results

Phase I: Preparatory work

Of the seven initiatives examined, the WHO Model Disability Survey, WHO Disability Action Plan, and Zero Project present disability-specific indicators. The other projects present general indicators. The most comprehensive coverage of the CRPD and the wide scope of CBR was provided by the MDS (n = 19), UNICEF MICS4 (n = 13), HDI (n = 12), Zero Project (n = 9), MDGs (n = 7), WHO Disability Action Plan (n = 6), and WHO Core Health Indicators (n = 5) (Table 1). The desirable outcomes of the CBR Guidelines covered 23 out of 26 selected CRPD articles.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Number of indicators from each project which were linked to the CRPD articles.

https://doi.org/10.1371/journal.pone.0178418.t001

Phase II: Revision of CBR desirable outcomes

Forty-eight of the 174 original desirable outcomes were eliminated for being dependent on specific objectives and activities of CBR, or for not being sensitive to changes at the person level in settings where CBR is carried out. In the components of education and livelihood it was found that some concepts reoccurred throughout the elements. In these cases, the cross-cutting concepts were formulated into single desirable outcomes which were moved to the general component level. For example, in the education component the concept of “Children, youth and adults with disability experiencing equal opportunities to participate in learning opportunities that meet their needs” reoccurs in all the elements of education, namely early childhood education, primary, secondary, non-formal education and lifelong learning. For this reason this concept was moved to the general component level. As a consequence, primary, secondary and non-formal education no longer had individual desirable outcomes. In livelihood, the concept of “People with disability earning income through their own chosen economic activities” reoccurs in the elements of self-employment and wage employment. These were moved to the general component level so that these elements no longer had individual desirable outcomes. Also within livelihood, the element of skills development had overlap with the lifelong learning component of education. For this reason it was decided to incorporate it into lifelong learning. The set agreed on at the end of the consensus process with IDDC consisted of 41 re-formed desirable outcomes (Fig 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The process beginning with the CBR Guidelines desirable outcomes and leading to the WHO recommended CBR indicators.

https://doi.org/10.1371/journal.pone.0178418.g002

Phase IV: Feasibility and validity testing

The total sample of the pilot implementations consisted of 801 participants, 53.4% female, with a mean age of 29.6 (SD 21.3). Further characteristics of participants are reported in Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Descriptive statistics of the 801 pilot implementation participants.

https://doi.org/10.1371/journal.pone.0178418.t002

Nineteen questions (37.3%) were reported as problematic in more than 10% of interviews, with five questions (9.8%) reported as problematic in more than 20% of interviews. Main problems reported by interviewers were that the question was confusing or difficult to understand, or that the question needed follow-up or clarification. Most problems were reported in Egypt (70.3%), followed by Guatemala (27.5%), and then China (2.2%). Focus groups in Guatemala, the first country carrying out the implementation, revealed problems that were then addressed before the pilots in Egypt and China. An overarching problem was the high complexity of some questions, and difficulties with the response options, which ranged from 5(completely) to 1(not at all). For this reason, response options were re-ordered from 1(not at all) to 5(completely), and show cards were used for visual representations in Egypt and China.

The expert survey invited 72 experts to participate, with 31 completing the survey. The majority of participants were male (54.8%) and worked in NGOs or INGOs (61.3%), while those who had worked in the field of CBR for 20 years or more (35.5%) and those working in the South-East Asia Region (38.7%) represented the largest proportions of respondents (Table 3). Per component and element, the indicator ranked as the most relevant by the majority of experts was selected. In cases where no indicator reached a majority vote, the top indicators were examined and selected by the CBR Group.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Demographic characteristics of the 31 expert survey participants.

https://doi.org/10.1371/journal.pone.0178418.t003

The analysis of the results of the survey and the pilot implementations led to the selection 40 CBR indicators[22]. The indicators are broken down into two sets: 13 base and 27 supplementary indicators. Base indicators are broad enough to capture the situation of people with disability, independent of specific CBR activities carried out and are derived from the general component level desirable outcomes. These indicators are recommended to be included in any data collection. All but one of the component level desirable outcomes was selected as a base indicator. The exception was the general livelihood indicator of “People with disability get to make their own decisions about how to use their money”. The CBR Group saw that the concept of “People with disability having enough money to meet their needs” was not covered by any indicator, and therefore created this as a base indicator, with the initial indicator remaining as a supplementary indicator. Supplementary indicators provide more specific coverage of the CBR elements and can be selected depending on the specific goals and strategies of a program. Base CBR indicators have eight corresponding questions and supplementary have 30 corresponding questions. The WHO manual presents the full set of indicators and the data collection procedures[22].

Discussion

The aim of this paper was to describe in detail all phases of the development process of a recently launched set of global CBR outcome indicators, based on the CBR Guidelines, which are suitable to monitor CBR. The proposed set of indicators includes 13 base and 27 supplementary CBR indicators, is grounded on the internationally acknowledged CBR Guidelines, and is the result of a collaborative, consensus-orientated and evidence-based effort between WHO, IDDC and the broader CBR community. These indicators will serve to capture the situation of people with disability in settings where CBR is carried out, independent of the specific objectives and implemented activities of a program. These indicators will support WHO and member states in their efforts towards strengthening CBR, as requested in the Global Disability Action Plan, through generating evidence on the effectiveness of CBR [6]. The use of the proposed CBR indicators will generate the evidence needed by NGO’s, DPO’s, and the broad community involved in CBR to advocate for broader and integrated CBR implementation in different settings, including at the national level.

The use of the CBR Guidelines as a multi-sectorial reference framework for the CBR indicators is essential. Due to the heterogeneity and varying contexts in which CBR is implemented, an appropriate framework is needed as a basis for the monitoring process[4,7]. The CBR Guidelines and the corresponding desirable outcomes were selected as a framework for the proposed CBR indicators as they encompass a unified understanding of CBR concepts in line with the CRPD[9,25]. Though a global set was previously suggested[13,26], there is some disagreement as to whether a global set of indicators, even when based on the CBR Guidelines and the corresponding matrix, can cover the cultural and methodological diversity of CBR[18,19]. To account for this, the CBR indicators proposed here take advantage of the several elements of each CBR Matrix component and use them to offer a possibility of customizing data collection. Stakeholders responsible for data collection are requested to use the 13 base CBR indicators in all data collection to guarantee standardisation and comparability. However, additional indicators can be selected out of the 27 supplementary CBR indicators so that the data collection can be shaped to monitor more specific programs’ goals, cultural settings, or requirements of funding bodies. In summary, the indicators presented in this project combine the advantage of providing a means of collecting global data for cross-program comparisons, while also addressing the diversity of CBR by allowing the flexibility to customize data collection.

The flexibility presented in indicator selection and the corresponding mobile phone app help to encourage the uptake of the CBR indicators by making data collection as quick and simple as possible. Providing intuitive procedures to customize and carry out data collection allows data collection to be carried out by any community member, which is in line with suggestions that the monitoring process should involve community members and people with disability to allow for engagement of the local community, thereby fostering greater community ownership and sustainability[9]. The app is free to download on Google Play and works offline. Interviews using base indicators can be completed within five minutes. Interviews are either submitted to a selected e-mail address or anonymously to a central and secure server located at WHO upon acceptance of the data protection agreement on the phone. Furthermore, in order to increase the motivation for data collection, if completed interviews are submitted to the central server, the data will be organized so that the indicator results can be presented as diagrams. These diagrams will be able to show the differences between people with disability and those without disability in the community surveyed, and within those groups, the differences between boys, girls, men and women. In case stakeholders are willing to share the data with WHO and the CBR community, anonymous comparisons of different programs and regions will be implemented in the CBR page of WHO’s website.

The CBR indicators proposed at present are the first necessary step towards the global monitoring and evaluation of CBR. They focus on monitoring and on outcomes at the individual level with the results from each indicator allowing for the identification of discrepancies experienced by people with disability. For example, when the indicator “% of people with disability who acquire education in mainstream education facilities” presents low percentage it can indicate exclusion of people with disability from their peers. These results can be further interpreted to see the effects on the community members as a whole. The next step in the monitoring process of CBR is to broaden the perspective by developing system indicators suitable to capture societal, administrative, attitudinal, and environment changes. Further work is also needed to develop sound and reliable indicators for the evaluation of CBR, in terms of creating making systematic judgements regarding the relevance, fulfilment of objectives, efficiency, effectiveness, impact, and sustainability of CBR[21]. As CBR is a continuous process there is an urgent need for longitudinal data to capture change over time, both for monitoring and evaluation, which will come through follow-up data collection.

Some limitations that come as an inherent result of using indicators, as is the case for the CBR indicators, should be mentioned here. People may argue that indicators have been shown to lead to over-aggregation and over simplification of data while only measuring what is quantifiable, and not always match to what is important to people[27]. Being unaware of this can lead to overconfidence in the relevance of the data collected, and thereby lead to incompleteness in the overview the indicators should generate. Furthermore, data gathered with indicators should be complemented with data from direct experience if an in-depth understanding is targeted, which can only be collected through qualitative approaches[27]. These facts might result in reluctance to use the CBR indicators. It is important to stress, however, that until now qualitative studies have dominated the field of CBR, and that despite all research carried out, recent reviews continue to stress the lack of evidence on the effectiveness of CBR[4]. In this sense, the proposed indicators may suffer from the inherent shortcomings of indicators, but they provide a unique opportunity to collect standardized global data on CBR after more than 30 years of attempts to do so. Data collected with the indicators, combined with results from available qualitative work, could finally prove what is strongly assumed, namely that CBR is effective and worth the effort required for implementation.

Finally, it is important to stress that the Global Disability Action Plan explicitly calls for the strengthening of CBR through monitoring and evaluation[6]. It is strongly recommend that qualitative work on disclosing potential sector, country, regional or cultural barriers, as usually done in implementation research, be carried out alongside the first implementations of the CBR indicators. Researchers and stakeholders are encouraged learn from data collection efforts and to contribute to the further development of strategies that can guarantee uptake of the CBR indicators. Users of the CBR indicators proposed here are therefore called to be active participants in achieving this goal by periodically collecting data, by reporting their experiences during data collection and by sharing data with WHO and the CBR community. This will contribute to the creation of a strong evidence base that can ultimately deliver arguments to improve CBR and potentially advocate for broader and more sustainable implementation.

Conclusion

The use of the CBR indicators proposed in this work and corresponding questions allow for reliable, easy and comparable data collection to demonstrate the effect of CBR, and thereby potentially broaden the appeal for its implementation. These indicators capture the situation of people with disability in comparison to other community members in the aspects of health, education, social life, livelihood and empowerment, as outlined in the CBR Guidelines[15]. When data is collected over time in a community it will capture changes in the lives of people with disability, as well as support monitoring of the implementation of the CRPD at the community level in an easy and efficient way. These indicators allow for further comparability across settings and countries. The CBR indicators are understood as a starting point towards generating sound and standardised evidence for CBR. Further work is needed to complement these indicators with system level indicators tackling factors in the environment, to identify barriers that might prevent their uptake, and to develop methods of using the generated information in economic evaluations of CBR.

Supporting information

Acknowledgments

Disclaimer: The views expressed in this manuscript are those of the authors and do not necessarily represent views or policies of the World Health Organization.

The authors would like to thank the International Disability and Development Consortium (IDDC) Community-Based Rehabilitation (CBR) taskforce who attended the CBR Indicators Workshop, held in Geneva on 9–11 February 2015, and who provided ongoing consultation throughout the development of the indicators and the corresponding survey questions: Marieke Boersma (Light for the World), Svein Brodtkorb (Norwegian Association of Disabled), Priscille Geiser (Handicap International), Karen Heinicke-Motsch (CBM), Aidan Leavy (Plan International), Antony Sahayarani (German Leprosy and Tuberculosis Relief Association), Evert Veldman (Enablement), Mary Wickenden (Institute for Global Health, University College London), and Roelie Wolting (Dutch Coalition on Disability and Development).

The authors express gratitude to those who lent their knowledge and experience in the preliminary expert surveys and to those who conducted the pilot implementations: Egypt- Nabil Ezzat Halim and Kerellos Kamel Saleb (Coptic Evangelical Organization for Social Services), and Gehan Fame Gendy, Heba Roshdy Kamel, Bassem Agaiby Samuel, Nesreen Shehata Sakre, and Hamdy Kamel Shehata (Elforssan DPO/Future Association); China—Sheng Cai (You and Me Community); Guatemala- Gonna Rota and Francisco Sojuel (ADISA program), and Oralia Méndez, Anelby Mogollón, Josúe Tzunun and Marí de León Xicay.