Development of multi-criteria decision analysis (MCDA) framework for off-patent pharmaceuticals – an application on improving tender decision making in Indonesia

Off-patent pharmaceuticals (OPPs) are among the most widely used health care technologies in many diseases with a large impact on public health. As first line treatments for many chronic conditions [1‐3], the success of public health policies in improving the health status of total population highly depends on the success of OPP policies, especially in developing countries where health care resources are even more limited to cover significant unmet need [4]. The lowest price is often considered the main efficiency criterion of OPP policies by health care decision makers. While this approach assumes that OPPs (comprising off-patent originals, branded generics and unbranded generics) are associated with the same attributes as their reference products in terms of quality, real world effectiveness or stability, there are shortcomings even in developed countries with high regulation standards [5]. Obviously, even more concerns could be raised against this policy objective in developing countries with a less highly regulated pharmaceutical system.

In the developed countries, bioequivalence (BE) has been fully adapted as a requirement for generics. However, in most developing countries even pharmaceutical equivalence (PE) has not yet been implemented for copies of original OPPs. Similarly, while in developed countries, Good Manufacturing Practice (GMP) according to EU/ Pharmaceutical Inspection Co-operation Scheme (PIC/S) standards including regular audits is considered to be a guarantee of constantly high product quality, in several developing countries even the lower WHO GMP standard has not yet been adapted [5]. Capacity problems of manufacturers, lack of profitability, inefficiencies of regulatory processes or pharmaceutical tendering, increasing demand, quality issues, unstable political or economic environments and inefficient distribution systems are among the concerns which are even more specific to increasing drug shortages not only in developing countries but also in developed European countries [6, 7]. These factors necessitate the consideration of multiple criteria in drug decision making especially in developing countries.

Indonesia has been using the Universal Health Coverage (Jaminan Kesehatan Nasional, JKN) since 2014, wherein the e-Purchasing procedure has been established for drug procurement for the national formulary. The drug prices and volumes to be annually utilized for the JKN program are listed in the E-Catalogue system by the National Commitment Officer (Pejabat Pembuat Komitmen, PPK). This procurement data is included in the National Procurement Portal (later known as Drug Procurement List) which is classified into single and multiple source (OPP) products. Single source products are evaluated by a committee through negotiation processes and multiple source products through public tenders. Tender evaluations are determined by price and non-price criteria (for example, quality and supply capacity) [8, 9]. However, the limited consideration of critical parameters in the current methodology, including failures in hospital estimations on the required drugs calculation (RKO), E-Catalogue and manual procurement as well as variation of drug distribution processes all lead to the nationwide pharmaceutical supply shortages for several drugs [10, 11].

A Multi-Criteria Decision Analysis (MCDA) framework could support the decision-making process by aggregating multiple policy objectives [12]. It is a methodology for appraising alternatives on individual, often conflicting criteria, and combining them into one overall appraisal [13, 14]. Each criterion considered in MCDA has a relative weight within the final evaluation reflecting its relative importance within the decision context. The subject of the evaluation is scored according to how it performs within each criterion. The aggregation of the scores received for each criterion multiplied by their relative weights results in a composite score. The scores of each alternative option can be compared to improve the objectivity, transparency and consistency of decision making. This tool has been increasingly used in health care to facilitate a wide spectrum of decision problems [15, 16]. For example, application of MCDA in the off-patent sector is now considered in China to inform Beijing tendering decisions and Egypt for vaccine tenders [17, 18].

In order to improve the evidence base of theoretical and practical implementation levels of making decisions in the off-patent sector in developing countries, an International Outcomes Research Board (IORB) including experts from academic, public and private sectors was established and held multiple workshops. One of the key deliverables of these workshops was the MCDA ‘Simple Scoring’ tool adapted for off-patent products to overcome one frequently mentioned barrier of MCDA, which is, that it is too complex for local decision makers to use, especially without previous experience. The Board identified a list of 22 criteria that could potentially be used to evaluate and make multiple policy decisions about off-patent medicines including their pricing, reimbursement, formulary listing, drug procurement, prescription, authorization and research. The 22 criteria, presented on September 6th, 2016 in the ISPOR MCDA Simple Scoring Educational Forum workshop in Singapore, represents a solid and scientifically rigorous basis for off-patent product evaluations which also allows for customization to the needs of developing countries [5].

The aim of this study was to develop, test and fine-tune a multi-criteria decision analysis framework on the basis of the MCDA ‘Simple Scoring’, designed to facilitate the decision-making process during the national procurement of off-patent pharmaceuticals in Indonesia.

Seven potential decision contexts were identified in the literature review for MCDA application of OPPs including market authorisation, pricing, coverage/reimbursement, national or hospital formulary listing and national procurement or hospital tenders [5]. The Indonesian national procurement of OPPs was considered to be a relevant decision problem to be improved with the use of MCDA methodology.

A five-step evidence-based approach was used to prepare the initial criteria list. The 22 items in the ‘Simple Scoring’ MCDA framework by Brixner et al. was the starting point for criteria selection [5]. The second step was to extend the scope of the ‘Simple Scoring’ framework with an additional literature review. The literature review identified 7 additional criteria. As a third step, in multiple teleconferences local academic experts from Universitas Gadjah Mada (UGM) provided an overview on current national procurement processes in Indonesia and indicated potential concerns with the current provision of OPPs. The objective of the fourth step was to reduce the criteria list to below 10–11 items in accordance with the recommendations of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR), to reduce the complexity, but not sophistication, of it for future use [12]. Based on local insights the 29-item criteria list was reduced by excluding several criteria 1) with limited relevance for national procurement of OPPs in Indonesia (e.g. budget impact, cost effectiveness, order of entry) and 2) without explicitly defined scoring functions (MCDA scores should be derived by clearly defined rules or functions aiming to convert performance measurements into scores). In addition, to meet the principles of completeness, non-redundancy, non-overlap and preference independence recommended by ISPOR [12], some of the remaining criteria were merged. For example, Pharmaceutical equivalence, Interchangeability and Bioequivalence were all merged into a new criterion called ‘Equivalence with the reference product’; and Technical assistance, Disease awareness & education and Continued Medical Education were merged into ‘Added value services related to the product’. In the end, the draft list of the MCDA framework contained 8 + 1 mutually exclusive criteria, including 1) Equivalence with the reference (original) product, 2) Real world clinical or economic outcomes such as adherence or non-drug costs, 3) Product stability and drug formulation, 4) Quality assurance, 5) Macroeconomic benefit, 6) Reliability of drug supply, 7) Pharmacovigilance, 8) Added value service related to the product and 9) Price. The initial criteria list, along with their intended definitions and corresponding performance categories are listed in Table 1. As the final step, the relevance of this initial criteria list was validated by local academic experts from UGM.

The selection of ranking and weighting methodology was based on the typology of the ISPOR MCDA Task Force [12] and the review of scientific literature on MCDA methodology. In an initial screening, the following ranking and weighting methods were excluded: 1) those with less of a solid theoretical foundation, 2) those which were not able to be implemented through a voting system and 3) those methods which were time consuming considering the duration of the 2-day on-site workshop. Those methods remaining on the short list after the initial feasibility screening were ranked according to their simplicity/implementation time and solidity of methodology. Finally, the modified Simple Multi-Attribute Rating Technique (SMART) method for ranking and swing weighting were selected based on the decision of an expert panel as an optimal balance between feasibility and scientific rigor [19, 20].

An MCDA tool was developed in MS-Excel, with spreadsheets containing all criteria, the corresponding performance categories for each criterion and their scoring functions. Criteria weights were to be entered based on the SMART and swing weighting voting results of the workshop. The performance matrix (main evaluation sheet) of the Excel tool was designed to be applicable for routine use when comparing multiple competing products in the national procurement process. The performance of each product according to different criteria can be selected from drop-down menus linked to the corresponding performance categories. The draft MCDA tool was designed to translate the performance matrix of each product to an aggregate MCDA score according to the scoring function and weight of each criterion, and indicate the ranking among competing alternative medicines.

The proposed methodology on ranking and weighting, the draft criteria list, initial performance categories and proposed scoring functions were all discussed with-, and validated and approved by academic experts from different universities in Indonesia during two teleconferences. These experts were also invited to the policy workshop. To facilitate knowledge transfer even before the workshop for those with more limited background information on MCDA, a customised pre-reading material was developed and circulated among workshop participants.

Additionally, a set of four competing test cases was developed and pre-calibrated to simulate real world tendering decisions. Each test case represented a full drug profile described in a one-page summary report, containing information on how the pharmaceutical product performed according to each criterion.

A 2-day on-site workshop was organized with the participation of key local stakeholders involved in the public procurement of off-patent pharmaceuticals. An overview of the policy landscape for procurement of OPPs was provided along with a background on the criteria for consideration in procurement decisions. The key objective of this workshop was to 1) select and approve the relevance of criteria in the Indonesian procurement process of OPPs 2) rank and weight the criteria 3) approve scoring functions of each criterion as well as 4) validate and fine-tune the MCDA framework based on reference cases.

According to a recently published guidance on the implementation of MCDA framework in developing countries [21], MCDA development should be started from the current decision-making criteria. Consequently, the first step was to determine the weight of pharmaceutical acquisition cost in the new MCDA framework. Workshop participants had to choose between values from 5 to 100% by 5% increments through anonymous voting implemented via a voting machine. The median value was calculated to reduce the impact of outliers. The second step was to determine the scoring function for acquisition cost (price). Participants were asked to vote on how much should be the excess price, above which they reward a product with 0% score for the Price criterion. This excess price was determined compared to the lowest-price option rewarded with maximum points for Price criterion. In this step, a + 200% excess price means that products with prices at least 200% higher (i.e. three times or more expensive) than the lowest price candidate will receive 0% of the scores according to the Price criterion (cut-off point), while products with a 100% higher price (i.e. twice as expensive) will receive 50% of the maximum scores that can be granted based on the Price criterion. The full 100% of price scores are to be awarded to the lowest price option. Therefore, the price function was linear and decreasing from the lowest price product (with 100% of the price score assigned) to the cut-off point (with 0% of the price score assigned). Participants could select the excess price (i.e. the cut-off point) between + 25% to + 500% to modify the steepness of a scoring function for prices and the median values were calculated to determine the cut-off point. Having a price scoring function independent of the price distribution of the competing products (i.e. by not linking the zero point for price to the most expensive product) prevents differences from being too large within the points awarded when product prices are in a small range or being too small in points awarded when the product prices are in a large interval. In other words, the steepness of the scoring function can be adjusted to the expected price distribution of the competing products.

As an introduction to the third step, participants were presented the 8-item list of non-price criteria, including 1) the definition of criteria, 2) the performance categories (see Table 1) and 3) the proposed scoring function for each criterion. The draft scoring function for each non-price criterion was defined by percentages of scores assigned to the corresponding ordinal-scale performance categories and was validated by local experts before the workshop. A higher percentage indicated better performance, with 100% being the full score that could be awarded any particular criterion, while 0% represented the worst score for performance of any criterion. Very poor performance in some key criteria resulted in their exclusion from the assessment (exclusion category). At this stage participants had the opportunity to add or delete items from the criteria list and the performance categories as well as to adjust the scoring function by majority voting. Once attributes of non-price criteria were discussed and agreed upon, as the third exercise participants were asked to rank the criteria based on their importance according to the modified SMART method. As a warming-up exercise, participants were presented a ‘worst case scenario’ (i.e. a hypothetical product performing in all the criteria at their worst levels). Participants were then asked to identify and vote on that criterion which they thought should be moved from the worst level to the best performance level to improve the performance of the hypothetical product. This was done to identify the most important criterion. Then participants were asked to vote on another criterion to be moved from the worst level to the best performance level again, to improve the performance of the hypothetical product. This identified the second most important criterion etc. In the end, the modified SMART process resulted in a list of the different criteria in order of their importance.

Once the ranking between criteria was established, a swing weighting method was used to estimate the relative weights of the non-price criteria as the fourth step. In the swing weighting, a fixed number of points (initially 10), was assigned to the least important criterion. Ten being the lowest point score, participants voted on how much more points the second least important criterion should be assigned to reflect its relative importance. Participants could select percentages ranging from 0% to + 50% by 5% increments and between + 50% to + 100% (i.e. two times more important) by 10% increments respectively, and the median values were considered. If, as an example, the median of the audience’s votes was + 50%, the second least important criterion received 10 + 10*50% = 15 points. In the next vote, participants had to make their judgement on the subsequent criterion in the priority list, considering the points assigned to the second least important criterion (e.g. 15) as a basis (etc.). The voting process was continued until each of the increasingly important criteria received progressively higher points. Scores for swing weighting can be seen in the Result section. In case of equal ranking, a 0% increment was assigned to indicate that a criterion had the same importance as the previous criterion and thus received the same point values. After assigning points to all criteria, first, the non-price criteria scores identified by participants were summarized and normalized to 100%; resulting in the relative weights for each non-price criterion. Once the relative weights of non-price criteria were elicited based on their progressively higher point values, they were normalized once again, yet this time taking the weight of the Price criterion into account as well, to obtain the final weighted values. Participants were shown the final calculated weights for the price and non-price criteria and then had the opportunity to adjust their opinion on any price weight during a second vote. This completed the draft MCDA framework.

Participants were then provided a set of 4 different test cases. These described the performance of 4 hypothetical drugs according to the criteria of the draft MCDA framework. Participants populated the draft MCDA framework with the test cases to prepare the performance matrix. Finally, as a fifth step, participants had the opportunity to adjust the framework with a special focus on the price weight and the scoring function for price (e.g. cut-off point). Preparation of the initial MCDA framework ended at this stage.

The weight of acquisition cost is one of the most crucial criteria in MCDAs use for the procurement of pharmaceuticals. There should be an optimal balance between price weights that are too high suggesting that no meaningful improvement is recommended in the system and weights that are too low suggesting it will receive little support by policy-makers or payers. The settled weight of price in the current procurement process of off-patent medicines was too high according to the majority of workshop participants. As a consequence, it was initially reduced from 70% (specified by the current legislation) to 50%. Nevertheless, even the 50% initial cut-off point that participants voted for also appeared to be too tight and rigorous after validating the initial draft MCDA tool with test cases. According to the linear scoring function, this means that products with more than a 50% higher acquisition cost than the lowest price candidate should be assigned 0% of scores for price, still resulting in an emphasis on price that is too dominant. Therefore the price weight was reduced to 40%. Test cases also revealed the need to adjust the scoring function of the price, and so the cut-off point was increased to + 100%. This indicates the preference of workshop participants to choose more price premium when it brings other benefits expressed in other criteria. Overall, the application of test cases improved the understanding of workshop participants on how MCDA works and what their opinion on weights and scoring functions means in real world decisions.

The Chinese Beijing MCDA framework [17, 18] for off-patent tenders assigns a maximum of 100 points for non-price criteria (50 for product quality criterion and 50 for manufacturer site criterion, respectively) and an additional 30 points for price. Therefore, Beijing applies an even lower weight for price of 23.1% compared to the 40% weight recommended in Indonesia. However, as the test cases highlighted, the weight of price also highly depends on the scoring function for the price criterion. Similarly to the conclusion of the Indonesian workshop participants, the Beijing MCDA process also nominates multiple (namely two) winners.

Table 5 describes key steps to the implementation of a repeated use MCDA. The first 6 steps of the MCDA development and dissemination are already completed by running the policy workshop in Jakarta. However, further steps are necessary to have the final MCDA up and running in routine use in Indonesia for making OPP procurement decisions.

The action plan highlighted the necessity of standardizing the formal requirements of the submission dossier prepared by pharmaceutical companies. During the workshop, the test cases were provided to the participants in an easy-to-follow one-page performance matrix (Table 1), containing how the drug performed according to each criterion. Once the initial MCDA is being routinely used, a similar template should be considered as a cover page for pharmaceutical submission dossiers. This means that pharmaceutical companies should indicate scores based on the best available and submitted evidence for their pharmaceutical products and then the MCDA committee should validate initial scores after critical appraisal of the submitted evidence. At first, materials submitted by manufacturers should be checked by the secretariat of the procurement agency in order to prepare the initial performance matrix for each competing alternative. Some of the inputs submitted by the manufacturers, e.g. the supply track record of manufacturers or data on equivalence to the original product may be partially or fully validated by specific regulatory bodies, such as the Ministry of Health or the Indonesian drug regulatory agency. Then, based on the initial validation of scores, the cover page of the dossier containing an assessment on how the product performs according to the MCDA criteria shall be forwarded to a committee for final approval of the scores before making a recommendation.

In general, uncertainty in MCDA models can be related to model inputs and structural uncertainty [12]. As our MCDA framework is developed for repeated use, the uncertainty in input data should be managed at the critical appraisal phase by the secretariat of the procurement agency. Structural uncertainty can be related to disagreement on the criteria selection, weighting method or scoring functions. Although workshop participants represented all key institutions in Indonesia, the overall number of participants who participated in the voting was relatively low, which is a potential limitation of our study and represents room for structural uncertainty. Therefore, after a one-year pilot phase authors propose that the MCDA framework should be revisited based on the initial experience. Once the necessary adjustments on weights or scoring functions are made in a similar policy workshop, the final MCDA is ready to be officially introduced in the decision-making process. However, considering the periodic review of the MCDA tool (e.g. in every 3 years) may further reduce the structural uncertainty.

Explicit and publicly available decision criteria can potentially further improve the transparency and accountability of the national procurement process in Indonesia. Additionally, the introduction of other criteria could create incentives for manufacturers to invest into additional data collection, improved supply chain reliability or better quality off-patent products which may ultimately result in more health gain for the society. The study confirmed the potential relevance of MCDA methodology and easy-to-use tools in the pharmaceutical policies of developing countries even for OPPs. Many of the learnings from this process in Indonesia can be applied to other emerging market countries as well.