Using concept mapping to develop a human rights based indicator framework to assess country efforts to strengthen rehabilitation provision and policy: the Rehabilitation System Diagnosis and Dialogue framework (RESYST)

In an effort to acknowledge the critical connections between rehabilitation services and the health system we use the term “health related rehabilitation” in the same way as in the context of Article 25 of the CRPD [2] to denote individualized, outcome focused healthcare services. Specifically we adopted the definition proposed by Meyer and colleagues [46], which takes into account HRR at the micro, meso and macro level with an emphasis on organizational and system characteristics of HRR services. Ideally, these services consist of scientifically sound and evidence based diagnostic, treatment and therapeutic activities as well as other interventions that are regulated and organized by the health sector. They are typically delivered in an organisational setting by a multidisciplinary team of properly trained and certified professionals (or a single therapist when appropriate).

Because rehabilitation, by definition, targets persons whose functions are limited as a result of illness, injury or chronic disease, it is very rare for a HRR intervention or service to benefit only persons with long term impairments and disabilities. Thus an approach to defining “persons with disabilities” for the purpose of monitoring HRR should be broad enough to account for the variety of disease groups that may benefit from HRR. To maintain consistency with the CRPD, “persons with disability” are understood in this research “to include those women, men, girls and boys with long-term physical, mental, intellectual or sensory impairments which, in interaction with various barriers, may hinder their full and effective participation in society on an equal basis with others” [2]. However, the characterization of “persons with disabilities” contained in the CRPD does not restrict coverage to particular persons; rather, it identifies persons with long-term physical, mental, intellectual and sensory disabilities as the main class to be protected. The reference to “include” in Article 1 of the Convention could therefore extend its application to all persons with functional limitations, i.e., those with short-term impairments or persons who are perceived to be part of such groups such as people with controllable non communicable diseases and episodic disabilities[47, 48]. This view is also reflected in WHO’s Global Action Plan on Disability 2014–2021 “Better health for all people with disabilities” which is informed by the principles of the CRPD and UHC, among others [5]. As with the earlier World Report on Disability, the WHO global action plan directs governments’ attention to the health and rehabilitation needs of people with disabilities asserting that this concrete focus will lead to faster and more sustainable improvements in rehabilitation services and thus bring about larger benefits for all people who may need rehabilitation in the context of UHC. By the same token, this research and its resulting product will be relevant to all those who are traditionally understood as disabled and face restrictions in everyday participation but also to clinical populations who experience limitations in functioning.

In accordance with our study protocol [49] we used an innovative form of structured conceptualization, colloquially referred to as group concept mapping, to develop the indicator framework which we call the Rehabilitation Systems Diagnosis and Dialogue (RESYST) framework. Group Concept mapping (GCM) is a mixed method that integrates sound qualitative group procedures (brainstorming, categorizing ideas, and assigning value ratings) with multivariate statistical analyses to help a group describe their ideas on the topic of interest and represent these ideas visually through a series of related maps [50]. It is considered a well-established method in public health research [51, 52] that has been applied successfully in the past to develop measurement and evaluation frameworks and tools [53]. Acknowledging the utility and value of GCM as a participatory approach that democratizes the process of knowledge co-production [54, 55], experts in rehabilitation research have recommended GCM as an appropriate technique of concept elicitation that helps generate useful insights from diverse stakeholders to guide the development of standardized measurement tools [56]. In addition, as a mixed method, GCM deploys advanced statistical techniques that allow data from multiple sources to be combined and analysed to produce a mental model that reflects the collective thinking of participants [57]. This feature makes GCM particularly useful given the objective of the study to develop a consensus understanding of rights based monitoring of rehabilitation policies and programmes.

GCM typically involves the following stages: preparation, including participants identification and recruitment; idea generation; structuring of ideas into piles of similar themes and rating them against predefined criteria; data representation; and group interpretation. All data collection and analysis procedures were performed with Concept Systems Global Max© [58] and were accomplished in the period between March 2015 – March 2017. The steps taken to achieve the goals of our study are explained in the text below:

The Ethics Commission for Northwest and Central Switzerland considered ethical approval not necessary. All participants were assured prior to engaging in this study of data confidentiality, informed of the voluntary nature of their participation, and of their possibility to withdraw at any time.

Fifty six individuals generated 275 initial ideas in response to the focus question requesting indicators that would be useful to monitor the HRR aspects of the CRPD. These ideas were consolidated in a final set of 107 unique statements by the research team (see Additional file 1). Subsequently, 44 completed the on-line sorting and/or rating tasks. Specifically, 37 participants completed the sorting task, 41 completed the importance rating and 39 the feasibility rating. Participant characteristics are presented in Table 1. Affiliations of individual experts who participated in the sorting and rating phase are shown in the Additional file 2.

To ensure the reliability of data analysis concept mapping guidelines recommend a number of 10–40 participants to complete the sorting activity [50]. In this study, 74% of those who agreed to participate in the sorting phase completed the sorting activity (N = 37), thus falling within the recommended range. Although 41 participants have originally completed the ratings, importance and feasibility ratings of 4 and 3 individuals respectively were excluded from the final analysis because of extreme response patterns and missing data. Finally, of the seven individuals who took part in the group interpretation, four have completed both sorting and rating activities.

MDS analysis of the sort data produced a two-dimensional point map with a stress value of 0.269¹ which was below the average stress value of 0.28 (SD=0.04, range: 0.17; 0.34) as estimated in a recent meta-analytic study of GCM projects [66]. Thus, the point map was considered as sufficiently reliable to proceed with further analyses. Application of hierarchical cluster analysis to the point map resulted in a preliminary framework solution of 11 clusters which was judged by the research team to provide sufficient detail and still yield substantially interpretable content within each cluster.

During the interpretation session, participants reviewed the 11 cluster solution and modified it as allowed to increase within cluster consistency of content and enhance the map’s overall interpretability. For example, the boundaries of the cluster labelled Service Coverage, Utilization and Outcomes were redrawn to include also Indicator #31 (see Additional file 1), which, initially belonged to the Social Mobilization and Research cluster, but was perceived by participants to be more conceptually related with service delivery outcomes. Similarly, the cluster Workforce Development was reshaped to include Indicator #1 (see Additional file 1) from the Evidence informed and Rights based Programming cluster, as participants felt it made more sense to be grouped with the indicators that pertain to workforce issues. It is important to note that these changes aimed at improving the overall interpretability of the map and did not alter the underlying data structure (point map) which reflects how participants sorted the indicators.

The final conceptual structure that emerged from the analysis and interpretation of the sorting information organized the 107 indicators into 11 non overlapping clusters as shown in Fig. 1. Experts who participated in the interpretation session confirmed the appropriateness of the cluster labels. Each cluster is described below in descending order of average importance. The full list of indicators within each cluster of the RESYST framework is presented in the Additional file 1.

Monitoring and Accountability contained 7 indicators focusing on the existence of information systems to track rehabilitation resources (Indicators #37, #52) and the availability and use of intelligence to ensure evidence informed management and public accountability (Indicators #48, #58). Some indicators however were thought to be conceptually related with ideas contained in adjacent clusters (Indicators #53, #54) as shown by their mid-range bringing values (see Additional file 1).

Access Barriers contained 12 indicators capturing structural (Indicators #60, #78, #92), organizational (Indicators #16, #46, #55, #74, #87, #101) and financial (Indicators #46, #82) barriers to access to HRR and assistive devices. This cluster included the indicator rated highest for importance across the entire set of 107 indicators (Self-reported barriers to access to medical rehabilitation [Indicator #46], see Additional file 1).

The cluster Service Coverage, Utilization and Outcomes was comprised of 14 indicators and had a comparatively low bridging value of .37. This cluster contained the indicator “Unmet needs for medical rehabilitation” (Indicator #65, see Additional file 1) which was rated second highest for importance across the whole set of indicators.

Rating information captured how important and feasible experts who took part in this study perceived the indicators to be. Indicator rating scores have been averaged per cluster (Additional file 1) and are also presented in the form of a ladder graph in Fig. 2. The overall mean rating for importance per cluster ranged from 3.15 (SD = .87, 95% CI: 3.08; 3.22) for the cluster Legal Commitments and Strategic Priorities to 2.23 (SD = .57, 95% CI: 2.09; 2.37) for the cluster Social Mobilization and Research. On average, the Legal Commitments and Strategic Priorities, which was perceived as the most important, achieved the highest rating score for feasibility whereas indicators contained in the cluster Access Barriers were on average perceived as the least feasible to implement.

Intergroup comparisons of importance and feasibility ratings were conducted to identify patterns of convergence and divergence in the views of participants with differing sociodemographic characteristics. The results showed that there was no significant variation in the responses and most participants followed similar patterns in prioritizing the indicators. For example, the results showed that there was a high degree of consensus between the group of “Rehabilitation professionals and/or academic researchers” and participants belonging to all other groups on both importance (r = .86, p < 0.01) and feasibility (r = .95, p < 0.01) ratings. Similar patterns were found in additional analyses with intergroup correlations ranging from .69 (p < 0.05) to .98 (p < 0.01).

Figure 3 presents a bivariate plot mapping average importance and feasibility ratings for 107 indicators. This plot helped participants during the interpretation session examine the relationship between importance and feasibility at the item level and derive a two-tier set of priority indicators for further field testing.

Overall, 59 brainstormed statements were rated above average for importance and comprised the priority set of human rights indicators for rehabilitation (Table 2). Of those 36 indicators achieved above average score for both importance and feasibility and were perceived by participants as having higher potential for success in monitoring the implementation of the CRPD (Tier 1: Implementation priority, see Fig. 3). Two thirds (n = 24) of the indicators in Tier 1 arose from the Governance and Leadership domain with the majority (n = 13) belonging to the Legal Commitments and Strategic Priorities cluster. Conversely, 23 indicators obtained above average rating for importance but below average for feasibility (Tier 2: Development priority set, see Fig. 3) which, according to the participants, represent areas where investments need to be made to systematize and improve the collection of data so as to enable the effective review and assessment of the rehabilitation sector in the future. Interestingly, the priority indicators of the Workforce Planning and Performance cluster (see Table 2) were seen as less feasible to implement in comparison with other indicators within this cluster.

The 107 indicators of the RESYST framework were mapped onto the CRPD. Table 3 shows the relevance of the indicators generated from concept mapping for monitoring human rights norms and standards that are implicated in HRR and recognized in the CRPD. All 107 address directly or indirectly fundamental political and social rights in relation to HRR. The majority of the indicators cover the right to health and rehabilitation as expressed in Articles 25 and 26 whereas a large number of indicators capture States efforts to promote the implementation of the CRPD by raising awareness of disability rights among rehabilitation workers and promoting their professional development.

The framework described in this paper has important policy implications. Firstly, the RESYST and its content constitute useful tools for the successful implementation of the WHO’s Global Action Plan on Disability, under which governments are responsible to “provide health sector support for monitoring and evaluating the implementation of health policies to ensure compliance with the provisions of the Convention on the Rights of Persons with Disabilities” [5][p.10] and to “undertake situation analysis to inform policies and planning” [5][p.17] for rehabilitation. Also, the RESYST provides the preliminary evidence base to support the implementation of WHO’s recommendation “to assess existing policies, systems, services and regulatory mechanisms, identifying gaps and priorities to improve provision” [4][p.122] of rehabilitation. The clusters and indicators offer a data driven basis for Ministries of Health to develop sector specific reporting tools, including country templates, that can guide the collection and analysis of pertinent information and provide a rapid indication of progress towards the implementation of the Action Plan and other widely agreed policy recommendations. Such structured templates and instruments have been developed and applied successfully in the context of mental health [104‐106] and it is believed that replication of such an approach will impact positively the rehabilitation sector. Additionally, the indicators have the potential to inform the development of standardized data collection tools and resources by public health analysts and researchers, such as structured questionnaires, surveys and key informant interview guides and thus contribute to the harmonization of data collection practices in rehabilitation services and policy research. Populating the indicators with reliable and comparable data at regular intervals in the future may enable the comparative and longitudinal benchmarking of rehabilitation services and help decision makers draw more meaningful conclusions based on sophisticated country level analyses of the rehabilitation sector.

There were methodological issues that may have impacted on our results and lessons learned will help adjust the design of similar studies in the future. Despite the implementation of evidence based measures to increase participation in our study (personalized email invitations, and frequent personalized reminders) participation rates were lower than the average concept-mapping project [66], for both importance and feasibility rating, meaning that our results may be prone to self-selection bias. Although the relatively low number of raters maybe a weakness, experience from the implementation of the GCM methodology suggests that the number of generated statements, the number of clusters and most importantly the number of sorters in our study were well within the limits of what is perceived as a standard to obtain reliable results. [66] Also, as is the case with all studies that employ non probability sampling techniques, the specific results of this study may have limited sample-to-population generalizability.

This research had a strong focus on the rehabilitation related dimensions of the right to health of persons with disabilities. We have therefore adopted a practical definition for HRR to facilitate a common understanding of the programmes and services that stakeholders think of as “health related rehabilitation”. This definition, along with evidence generated from a previous legal analysis of States obligations under the CRPD, helped participants identify potential measures that address several factors that impact on access to services and the enjoyment of disabled people’s right to health. This emphasis on the health care aspects of rehabilitation may have biased our selection of potential eligible participants in favour of those who are more active in the health domain. This has resulted in an overrepresentation of rehabilitation professionals and disability researchers in the group of sorters and raters. Additionally, given the general scepticism over contemporary approaches and definitions of rehabilitation by the disability community, and the subsequent lack of trust in rehabilitation research endeavours, it is possible that the emphasis on HRR has deterred some representatives of people with disabilities from participating in the sorting and rating of indicators. However our decision to concentrate on these aspects is justified by the widespread and often insurmountable barriers people with disabilities confront in accessing rehabilitation treatment services. It could be argued that an approach that would place a greater focus on social aspects of rehabilitation, especially CBR, may have created a more enabling environment and provided greater opportunities for persons with disabilities - especially those from low and middle income countries - to express their voice on this important topic and may have allowed to capture a small but significant segment of rehabilitation service systems in low income countries that is disproportionately focused on community inclusive development. Given however the expressed concerns over the ability of CBR as a strategy to bring broader systemic changes in line with the CRPD and remove barriers in access to services [107], it is doubtful whether this would be the right choice. Even so, it would have resulted in an entirely different set of indicators that would not have been bias free either.

Notably, experience from research shows that the issue of participation of service users or community members in the development of measures and indicators is highly controversial (as is the definition of stakeholders in community based research) as there are no specific guidelines regarding participation rates and the optimal balance between stakeholders groups [108, 109]. This lack of guidance partly explains the low to moderate levels of participation of patients and service users in community engaged concept mapping research on health systems, where the majority of studies examined in systematic review appeared to adopt a consultative rather a collaborative approach to community participation [55]. Informed by a human rights approach to disability, our study set a goal to create conditions that would allow bi-directional input and collaboration with persons with disabilities who have traditionally been neglected in the discussions about accountability and monitoring. [49] We paid particular attention to achieving a balance in the recruitment of both advisory committee members and individual participants to avoid creating hierarchies of power that would undermine the contributions of persons with disabilities by extending an invitation to the same number of individuals that were believed to belong to the same stakeholder group. Despite these efforts, several exogenous factors have impacted on participation rates. In addition to the lack of willingness discussed above, other commonly reported factors that may explain the low participation rates of persons with disabilities and policy makers include, among others, the labor intensive process of data collection, the high cognitive load induced by the high number of indicators and the lack of time [55]. Future research with larger sample sizes, including oversampling for specific stakeholder groups (policy makers and consumers) and a combination of web based and face to face data collection is recommended to counterbalance these limitations.

Interpretation of the conceptual map took place with a convenience sample of experts who participated in the sorting phase coming mainly from high-income countries. The lack of involvement of participants from low- and middle-income countries in the group interpretation may have introduced a culture bias as it is possible that the results would have evolved differently if their insights on the grouping of clusters into thematically related domains have been incorporated into the final model. Discussion of the cluster map and the indicator set with a larger number of participants - especially policy decision makers and service users –is recommended as part of future local adaptation and/or validation processes, as this may enhance their relevance for implementation as well as the sense of ownership of the results of the monitoring activity.

Although the framework incorporates a large number of factors previously found to be important in assessing progress in the implementation of the HRR aspects of the CRPD, it is possible that some indicators that are standardly used in rehabilitation service evaluations and quality assessments may have been missed. This was expected as the emphasis of this research was on human rights indicators intended to complement, rather than replace existing performance measures. It is likely that with further research, including testing and validation of the framework considering local context and stakeholder priorities, additional indicators may be included in the framework.

The RESYST presents both similarities and differences with the EQUIFRAME, a policy analysis framework that has been previously developed to facilitate the examination of the compatibility of health policies with human right guarantees for vulnerable populations [110], including in rehabilitation [44]. Given the both frameworks used human treaties as the normative basis for their development, it is not surprising that many of our indicators, especially indicators of governance, reflect core concepts featured in the EQUIFRAME. Unlike, however, EQUIFRAME which has a broader focus on health policies for marginalized populations, our framework focused particularly on HRR for people with disabilities. As a sector-specific framework, the RESYST is more extensive and more comprehensive: it encompasses measures that capture all aspects of the rehabilitation related dimensions of the CRPD, including quantitative indicators of service availability, service utilization and outcomes. Our framework also makes a distinction between CBR and HRR. Methodologically, the process of development of the RESYST was statistically more robust and resulted in a mental model in the form of a visual map that reflects the complexity and systemic nature of rehabilitation service organization noted also by others [71, 72].

It will be important to test the validity and practical utility of the indicators in different contexts, assess the methodological constraints in data collection as well as the organizational costs and benefits associated with the use of the proposed indicators. This will require the drafting of indicator specification sheets (containing definition, rationale, method of calculation, interpretation and limitations for each indicator) and the development of multi-item assessment instruments to facilitate the collection, verification and analysis of information. The use of empirically developed and field tested indicators will provide opportunities to appraise rehabilitation policies and compare service organization across countries and thus move the scientific evidence base on comparative rehabilitation systems research forward. The practical insights offered by this study are both timely and strategically relevant as leading health agencies and professional organizations strive to integrate rehabilitation in health systems through capacity building and assessment initiatives. It is therefore recommended that health agencies, especially the WHO, professional organizations and international research consortia use the RESYST framework as an evidence source in future projects aiming to develop service monitoring and capacity assessment tools as well as to stimulate debate on methodological issues pertaining to the construction of system level indicators for rehabilitation.