The challenge of obtaining information necessary for multi-criteria decision analysis implementation: the case of physiotherapy services in Canada

As fiscal constraints dominate health policy and planning discussions both across Canada and globally, priority-setting exercises are becoming more common to guide the difficult choices that must be made [1]. In this context, it is not only appropriate but also highly desirable to assess the value of specific health care services, as an assessment of value is necessary for priority setting on resource allocation either through the use of a threshold (minimum value per dollar spent) or through a formalized priority-setting process such as Program Budgeting and Marginal Analysis (PBMA).

A common approach to the assessment of the value of health care services is economic evaluation [2]. Economic evaluation is typically used in a threshold approach to resource allocation, meaning that interventions costing less than a threshold cost per unit of benefit are deemed worthy of funding. However, there are well-known challenges to the acceptability of economic evaluation as a tool to guide resource allocation decisions. A key such challenge is to “ensure alignment between the objectives assumed in economic analyses and the objectives facing decision-makers in reality” [3]. Specifically, economic evaluation as a priority setting tool assumes that the decision-maker’s objective is to maximize health gain [4] but we know that other objectives are also typically pursued [4, 5]. One solution offered is that “the simple C/E ratio could be supplemented by information on other health effects for the patient, for example a descriptive account of expected improvements in quality of life; wider societal effects of the intervention, for example on the number of jobs created; and nonmonetary costs for the patient reported in natural units such as waiting time in days” [4]. Such a solution can in fact be formalized through the use of Multi-Criteria Decision Analysis (MCDA): “MCDA is aimed at supporting decision makers faced with evaluating alternatives, taking into account multiple, and often conflictive (sic), criteria” [6]. The criteria in MCDA are the ‘other health effects’, the ‘wider societal effects’ and the ‘nonmonetary costs’ referred to above, or put simply, the considerations that a decision-maker will typically take into account in making a decision on resource allocation. MCDA is typically used in formal priority setting processes such as Program Budgeting and Marginal Analysis (PBMA). Like economic evaluation, MCDA has methodological challenges, but in many contexts, because it formally includes most or all considerations relevant to decision-making, this approach, and the associated priority setting frameworks, fit the decision-maker’s perspective better [7, 8]. One key methodological challenge of MCDA is the search for the necessary information. The necessary information is often not readily available for two main reasons. First, some of the criteria, while relevant to decision-makers, are typically not common research subjects. This would include criteria such as integration or access. Second, even when literature is available, the information must be contextualized before it can be used. This paper reports on an example of how this key challenge can be addressed.

In the Fall 2011, the Canadian Physiotherapy Association (CPA) embarked on a project aimed at providing a valuation of physiotherapy services that is both evidence-based and relevant to resource allocation decisions in health care organizations. This project originated more than one year earlier, in 2010, when the CPA Branch Presidents concluded that there was a need for more information on the value of physiotherapy services and asked the CPA national staff to consider developing a document that would address this need. After investigating the methodological alternatives, the CPA national staff decided to proceed with the MCDA framework.

In this paper we report on the methods used to obtain information necessary for MCDA implementation and provide a brief summary of the information that was produced. The objective is to show how a key challenge to the implementation of MCDA can be addressed and give a sample of the results. The full results which are in the final report are the starting point for an MCDA implementation. Actual examples of full implementation of MCDA within a priority setting process at the local level, building on the information produced in this project, are not included.

MCDA involves the assessment of alternative actions on the basis of a common set of criteria [9, 10]. The two key elements of the MCDA process are the alternatives to be considered and the criteria to be used. Possible alternatives are those options available to the decision-maker, for example changing the level of funding for a given physiotherapy service or program. The criteria represent the relevant considerations in assessing the impact of implementing any of the different alternatives. Criteria therefore depend on the decision-making context. Once possible alternatives have been evaluated on the basis of the selected criteria, they can be compared and recommendations can be formulated. The evaluation of each alternative provides an assessment of what would be lost, in cases of a reduction in funding, and what would be gained, in case on increases in funding. When contextualized, this valuation represents the marginal value of a service at the local level (as opposed to the total or average value) as the question that was posed with respect to each criterion was: what would be the impact on this criterion of an increase or a decrease to the current volume of service. The basic steps in MCDA are outlined in Table 1.

The first step in the application of MCDA to the valuation of physiotherapy services was to determine a set of criteria relevant to decision making on health care resource allocation involving such services. The perspective adopted for this project was that of a decision maker within a health region or health service delivery organization, as this was the primary target audience for this work. Based on previous priority setting work with Canadian health authorities and on the literature on priority setting [11‐14] an initial set of criteria was proposed to the CPA and, through discussion, a final list of eleven criteria was developed (see Table 2). The criteria were defined in such a way as to ensure that overlap was minimized [i.e., they are meant to be mutually exclusive, as much as possible].

Moving to step two, the CPA identified a set of service areas for assessment, based on relevant literature and similar briefings on value for money developed in the United Kingdom by the Chartered Society of Physiotherapy. The final list contained twelve service areas (see Table 3).

This research project was about the second part of step two which is to obtain information necessary to assess each alternative on the basis of each criterion. This was done in a three part process. First, a literature review was undertaken to identify peer reviewed papers that address the notion of value (as defined by the criteria). Search terms included the given service area along with ‘effectiveness’, ‘cost-effectiveness’, ‘value’ and a myriad of other terms relating to the identified criteria. Due to the breadth of the search, a systematic review was not attempted; rather key papers were identified and reviewed with the intent of providing insight into a given service area, as opposed to a comprehensive take on each area. Not surprisingly, for many of the criteria there was no, or very limited, research evidence. The second part of the search for information or evidence was a series of interviews with content experts for each of the twelve service areas. These content experts (n=1 to 3 depending on the service area) were identified by the CPA. Through one or more phone consultations, the literature review for each service area was critiqued and new information was generated where no, or insufficient, literature existed. This was an important part of the process as the literature only provided information on some of the criteria. The missing, but required, pieces of information thus came from expert opinion. It is in this combining of expert opinion with research finding that MCDA provides a pragmatic approach to valuation.

After drafting of an initial synthesis document by service area which combined the results of the literature review with expert opinion, there remained a need to ensure that 1) the information presented would be applicable at a national level (as opposed to the provincial or regional level) and 2) the assessments would be consistent across the service areas (noting that the content experts were only focusing on a single service area). For this purpose, in the third part of the process to acquire the required information, the CPA struck a validation committee comprised of eleven individuals from across Canada with a broad range of experience in physiotherapy. Over the course of 2 two-hour meetings, the synthesis document was reviewed in detail. In some cases the validation committee requested additional information from the literature and clarification of points made by the content experts. The synthesis document was then adjusted to reflect the comments from the validation committee, including additional research information and clarification of expert opinions, resulting in the final synthesis by service area. Steps three and four of the MCDA process were not included in this project as those steps are context-dependant by nature.

In this section, we present the some of the key findings by service area. This section is limited to select key findings because full presentation is beyond the scope of this paper: we have 11 criteria and 12 service areas which means 132 cells of information which in the final report represented 55 pages of content. Where findings are based on published evidence, references are provided. When there is no reference, the findings are expert opinion, obtained as described above. Further details, including the key findings for each criteria, for each of the twelve service areas are presented in Table 4 (there again, where findings are from the literature, references are cited). Complete results can be found in the CPA report ‘Valuing Physiotherapy Services’ [15]. The findings presented here represent the minimum starting point required to implement MCDA at a local level. In the Discussion, we describe how these results can be used in an MCDA exercise in a health service organization.

In the context of choices that must be made because not all activities can be carried on as they were due to financial restrictions, information about the value of any given intervention is very useful [100]. A common framework for generating such information is economic evaluation where the cost per Quality-Adjusted-Life-Year (QALY) gained through a given intervention is estimated. The estimated cost per QALY gained however only addresses the impact on the life expectancy and on the quality of life of the clients or patients. In making decisions about allocating limited funding, decision-makers typically consider other objectives in addition to the direct health impact, with equity and access, for example, being often cited [101]. Moreover, economic evaluation is focused on specific end-points which are typically directly related to the condition, or potential condition, being addressed, for example, the extent to which physiotherapy services would impact a specific measurement of the progress of juvenile idiopathic arthritis. Because of these limitations, when the CPA decided to address what they felt was a gap in the available information on the value of physiotherapy services, it made the decision to address this gap through the application of MCDA. The overarching thinking behind this decision was that, as healthcare organizations face increasingly tougher choices, the limitations of QALYs as a resource allocation tool will push organizations toward more formal resource allocation frameworks that use MCDA in their evaluation of alternatives and physiotherapy services will be more likely to receive fair consideration if the health care organizations have access to accurate information. PBMA would be one of these frameworks. The choice of MCDA was not primarily guided by the relative level of difficulty in implementing a QALY approach versus MCDA. There was what was perceived as a shortcoming in information on the value of physiotherapy services and it was decided to put effort in the MCDA approach provided a greater potential for impact.

The result was a comprehensive report summarizing the value of each of twelve areas of physiotherapy services with respect to each of eleven criteria that were thought to represent all relevant considerations in making decisions about funding involving those services. Some key findings in terms of benefits of physiotherapy services are presented in this paper. It must be recognized that this paper is not reporting on an implementation of MCDA, or of a prioritization exercise. In fact, what the CPA has done is supply health care organizations in Canada and elsewhere with a base of research work necessary for the implementation of MCDA, as part of a resource allocation framework such as PBMA. The findings can be used as a starting point within any local MCDA implementation. It is not the role of the CPA to contextualize the information, assign weights to the criteria, or even suggest that only the criteria listed here should be used, or to actually rate the impact of service volume changes. These steps are the responsibility of local health care organizations. An organization that decides to implement the MCDA framework to guide resource allocation would have to: 1) determine locally relevant criteria and weight them (these could be different than the criteria used in this study but it is not expected that there would be significant differences); 2) identify possible service volume change options that make sense in their context (which depends on the existing mix and volume of services provided); 3) assess the impact of each option on the basis of the selected criteria (this is where the information contained in the CPA report comes into play and provides a necessary starting point, i.e. necessary but not sufficient information). Note that the breakdown of areas of practice may not perfectly fit a given local context, in which case, the relevant areas from the twelve used here can be combined; and 4) rank the options and make decisions. All these steps are standard practice in most prioritization framework, and are part of the PBMA, for example.

Our objective here was not to provide one more case of PBMA implementation but to address a common criticism of PBMA or any other process that includes MCDA: that the required information is either not available or too difficult to obtain making such processes unimplementable and therefore only theoretical constructs. . In terms of information generation, the literature review posed no unusual challenge. As was expected, in the grid of criteria by service area, many of the cells were left blank after the review. The recruitment of experts was done by the CPA and didn’t seem very difficult for two reasons: this was a project of the CPA and many members are very supportive of their organization and it is a project that many members can relate to and specifically support. Furthermore, the demands are not overly burdensome as each expert was asked to participate in one or two calls of one to two hour each. What is more challenging is explaining to the experts what is needed from them which is to provide a response to the best of their knowledge and not limit their answers to what they know is research evidence- we really wanted their expert opinion. While this did not come naturally to some of the experts involved, all ended up contributing as was needed. Putting together a validation committee was no more challenging than recruiting experts for the same reasons. And just as was the case with the experts, it is necessary to have a full explanation of the process and some basic training in MCDA before the committee can start to work. The main challenge with validating the local data to the national level was understanding how much of the expert opinions were shaped by unique local circumstances. This was addressed by first identifying where this might be the case, going back to the local expert for further information, and then reconvening the validation committee. The key lessons from this experience were: 1) there has to be experts that buy into what is being done, reluctant participation would defeat the process; 2) explanation of the process and its goals and basic training is necessary before the experts can be asked questions. Finally, it must always be remembered that the ultimate goal is to obtain the best existing information, sometimes experts feel uncomfortable with expressing their opinion in response to a question but if it is the only available information then it becomes the best existing information. In our project, we found some initial hesitation in some cases but all experts were able to overcome it. The main limitation of this paper and the supporting report is the extent of the resources available for this project. For many criteria, the principal source of evidence was expert opinion and this was provided on a strictly voluntary basis. There was sufficient input into the process to produce validated results but, without a doubt, more resources would have produced a more refined report. However, a benefit of the MCDA approach is the transparent nature of the process which allows ongoing updating of the results. As new studies are published or as more experts can devote time to this analysis, findings can be continually updated, by area of service or by criterion. And further areas of service can be added.

As the growth in public health care funding slows, more difficult choices about what to fund and what not to fund must be made. In this context, relevant and accurate information about the marginal value of any health care interventions is essential for proper resource management. MCDA can be a very effective means of producing such valuations which can then be used in whatever priority setting process is implemented. However MCDA requires evidence on aspects of value where there is typically very little research evidence available. In this paper we have described an approach to addressing this challenge. The results presented are valuable for two reasons. First, a pragmatic approach to the generation of necessary evidence is presented. While this approach may seem rather obvious, the fact is MCDA and priority-setting processes that employ MCDA are often denigrated on the basis of the implied demands for information and the challenges that this poses. Second, this paper also provides a glimpse of the findings that were generated which may lead some readers to refer to the final report as a solid starting point for an application of MCDA involving any of the twelve areas of physiotherapy services studied.