1 Introduction

In 2007 the Government of Pernambuco, a State in the Northeast region of Brazil with about 8.8 million inhabitants, asked the various State Departments to prepare their programmes to the 2008–2011 Multi-Annual Plan (PPA—“Plano Plurianual”). The Head of the new State Department of Social Development and Human Rights (SEDSDH—“Secretaria de Desenvolvimento Social e Direitos Humanos”) saw this as an opportunity to gather the senior decision-makers and technical staff of the previously autonomous and heterogeneous entities that had been recently integrated into SEDSDH. His goal was to get them aligned and committed to SEDSDH strategic objectives and translate these objectives into agreed intervention programmes for the PPA. The social challenge was presented as “to formulate, execute, monitor and evaluate, along with society and other governmental entities, integrated public policies in the field of social development and human rights which will enable, in a conscientious and desired way, the transformation of the social reality of the Pernambucans living in a situation of vulnerability and risk” (SEDSDH 2007).

Such a setting—in which several stakeholders or representatives, who have different views and potentially opposed interests in the decision outcome, face the challenges of generating corporate strategic plans, reaching consensus or compromise, and prioritising complex issues—is typical in group support system applications (Lewis 2010). It is well-documented that groups encounter obstacles when attempting to work together to make decisions and solve problems (Lewis 2010) and decision support systems (DSS) have proven to play a key role in overcoming such difficulties by effectively assisting groups, namely in the generation and organization of ideas and in the use of voting to promote the achievement of consensus (Vogel and Coombes 2010). With these expectations, it was suggested that SEDSDH follow a socio-technical approach for group decision support, which builds upon lessons learned from several public strategic planning processes with the direct involvement of politicians [as in developing a strategic plan for the city of Barcelos (Bana e Costa et al. 2002)] and other actors [as in developing a vision for Puerto Rico where more than 150 stakeholders participated (Puerto Rico 2025 2004)]. By focusing on support systems aimed at facilitating a group in the structuring, analysis and negotiation of decisions, the proposed approach is within the scope of the facilitated group decision and negotiation branch of the group decision and negotiation literature (Kilgour and Eden 2010). It has the components of a group support system by integrating technology, a group of participants and facilitation (Ackermann et al. 2005).

The political actors agreed to participate in open discussion sessions together with technical actors. These sessions were decision conferences (Phillips 2007; Phillips and Bana e Costa 2007; Lewis 2010) developed under the principles of process consultation (Schein 1999), in such a way that the effects of preferences and choices taken by participants during the sessions were quickly reported in a friendly way, and so that those effects could be easily understood by all the participants, thus enabling collective learning and the generation and debate of new ideas (Bana e Costa et al. 2002). In short, a group learning strategic thinking process developed taking the view that, as well stated by Bryson (2004), “what matters most is strategic thinking, acting, and learning” (p. iii). Such a type of learning and collaborative process has also been highlighted as central for decision implementation (DeSanctis and Gallupe 1987), which was a primary concern of SEDSDH’s Head.

This article describes how the combined use of several techniques and methods of problem structuring, multiple criteria decision analysis (MCDA) and strategic thinking enabled the participants in the socio-technical process to: identify what should be the SEDSDH fundamental objectives (development axes); generate, assess and classify intervention programmes to achieve the objectives; and, to reach an agreement on which programmes should be proposed to integrate the 2008–2011 Multi-Annual Plan (PPA) of Pernambuco. More specifically, this multi-methodological framework (Mingers and Brocklesby 1997) integrated cognitive mapping and analysis of interconnected decision areas (Mingers and Rosenhead 2004), value-focused thinking (Keeney 1992) and multi-criteria value modelling (Belton and Stewart 2002) with MACBETH (Bana e Costa and Vansnick 1999), and strategic graph analysis (Matheson and Matheson 1998) and portfolio decision analysis (Salo et al. 2011) for project prioritisation. It is a “methodology enhancement” (Mingers and Brocklesby 1997) of MCDA with contributions of other decision support methodologies, “under a single paradigm—the learning paradigm—that guaranteed the theoretical and practical cohesion and consistency of the process at all stages of development (Midgley 1997)” (Bana e Costa et al. 2002, p. 435). Multi-modelling applications are not common in the group decision and negotiation literature and are a priority in the research agenda (Salo and Hämäläinen 2010). Although strategy workshops are common in formal strategic planning processes, most of them rely on discursive rather than on analytical approaches and MCDA is seldom used (Hodgkinson et al. 2006). This paper provides an empirical validation of the effectiveness of MCDA and decision conferencing to support group decision-making in a strategic workshop setting, if appropriately tailored (Montibeller and Franco 2011a). In particular, it suggests the enhancement of MACBETH with a voting procedure to ease the interactive generation of group compromise judgements about the desirability and doability of intervention programmes. It also shows how the concept of value-for-effort—which differs from the traditional value-for-money perspective (Phillips and Bana e Costa 2007)—is useful for programme prioritisation.

The context of the SEDSDH case is addressed at the beginning of Sect. 2, from which the analytical approach was tailored and the intervention process designed. Having in mind the importance of reporting how facilitators design collaboration processes (Kolfschoten et al. 2007), the second part of Sect. 2 offers an overview of the structure of the group decision support process. Sections 3–5 then offer a detailed description of the methods and procedures followed to structure objectives and programmes (Sect. 3), evaluate programme benefits (Sect. 4) and develop a value-for-effort prioritisation of the programmes and analyse its robustness in face of disagreements on the relative importance of improving the current situation of Pernambuco on the fundamental objectives (Sect. 5). Section 6 reflects upon the experience at SEDSDH, discusses issues related with the use of DSS in this study, as well as suggests topics for further research.

2 The Socio-technical Approach

2.1 Application Context and Decision Support Techniques

The proposed socio-technical approach for group decision support was detailed in a kick-off meeting with top managers of SEDSDH and discussed at the light of the Government directives to prepare the PPA (Governo do Estado de Pernambuco 2007), the “integrated model for social development and human rights” (SEDSDH 2007) and several questions that arose during the meeting. The most critical methodological challenge was to alter the subordination of objectives to programmes inherent to the Government “logic for the construction of a programme”: “Problem \(\rightarrow \) Programme \(\rightarrow \) Objective + Indicator” (Governo do Estado de Pernambuco 2007). A brief introduction to the principles of “value-focused thinking” (Keeney 1992) and its advantages over the “alternative-focused thinking” logic, led to a crucial agreement on reverting the sequence “Programme \(\rightarrow \) Objective” to: Objectives followed by Programmes to achieve them. Value-focused thinking also helped to make clear the difference and relationship between “strategic” objectives of the SEDSDH and “fundamental” objectives for the PPA (Keeney 2007). Under this methodological perspective, it was then possible to discuss the questions posed: How to articulate the actions of the eight administrative units of the SEDSDH and structure them into coherent programmes? How to define the PPA objectives? How to obtain more efficient programmes – greater synergy, avoid overlapping of actions/activities, duplication of effort, and “quixotic” and isolated actions? How to make the programmes more effective at achieving the objectives? How to assess the benefits (positive or negative contributions to achieve the objectives) of the programmes and the implications of their prioritisation? What is the best portfolio of programmes in the light of the objectives? How to involve the various actors in a participatory and structured process to achieve their commitment towards the resulting prioritisation?

Some of these questions bear on the problem of identifying, understanding and structuring key issues, for which several approaches have been proposed in the literature (see description in Bryson 2004). Among them, a causal mapping approach (Bryson et al. 2004; Eden 2004; Eden and Ackermann 1998) was followed, as interpreted in Bana e Costa et al. (1999b) and Bana e Costa et al. (2006), to support the identification of objectives and the definition of intervention programmes. Other questions are concerned with appraising the extent to which programmes achieve objectives. The strategy literature has given little attention to formal and explicit ex ante appraisal, in the eyes of the problem-owners and with their direct engagement, of the contribution of proposed actions to achieve the objectives. Or ad hoc evaluation procedures violating basic principles of decision analysis are suggested, such as weighted sums of ordinal ratings and direct importance weights (Thompson and Strickland 2003; Xu and Yeh 2012). To evaluate the programmes, an additive value model was constructed using MACBETH (Measuring Attractiveness by a Categorical Based Evaluation Technique), which has proven to be effective to avoid those common scoring and weighting pitfalls in a wide variety of real-world cases reported in the literature (Bana e Costa et al. 2012b). The combined use of causal mapping and MACBETH in strategic planning cases goes back to 1994 in the public context (Bana e Costa et al. 2002) and to 1996 in the private context (Bana e Costa et al. 1999a). An innovation in the SEDSDH’s case was the use of a voting procedure in the input stage of MACBETH value-elicitation, according to which the participants expressed individual qualitative judgements about the relative attractiveness of programmes. These judgements were subsequently discussed by the group, revised through an interactive and learning process, until compromise judgements were agreed by the group. Besides the evaluation of programmes’ benefits, an adequate answer to the question posed at the kick-off meeting regarding the prioritisation of programmes required also the appraisal of their doability, which also followed MACBETH voting. Ultimately, the concept of value-for-effort (being effort the complement of doability) was seen as more adequate for programmes’ prioritisation in the specific strategic setting than the traditional value-for-money approach previously used in more operational resource allocation contexts in local government (Bana e Costa et al. 2002) (see also Montibeller and Franco 2011b). The robustness of the selection made was analysed using PROBE, a multi-criteria DSS for portfolio robustness evaluation (Lourenço et al. 2012).

In short, a multi-modelling application was developed under an integrated perspective in which evaluation was taken as “an activity embedded in the planning process and supporting many other activities in that process” (Cerreta 2010, p. 383). Figure 1 presents a summary of the main decision support techniques and tools used at different stages of the socio-technical process: Decision Explorer (Banxia Software Limited 2002), AIDA (Friend and Hickling 2005), M-MACBETH (Bana e Costa et al. 2005) and PROBE (Lourenço et al. 2012).

Fig. 1
figure 1

Decision support stages and tools

Full size image

As said in Sect. 1, decision conferencing was the participative framework used to apply those techniques to facilitate “social interaction, mutual learning, and communication” (De Montis 2007, p. 473), in view of achieving “a shared understanding of the issues, a sense of common purpose and a mutual commitment to action” (Phillips and Phillips 1993, p. 533). This and other related social aspects relevant for model building are discussed in Sect. 2.2.

2.2 Designing the Social Process

The social process consisted of a series of decision conferences, designed at the start of the engagement to develop the strategic planning. The participants were thirty political and technical problem-owners covering the different perspectives within the SEDSDH. They were grouped into two work teams, the decision-making team (the “DM-team”) and the technical team (the “T-team”), respectively, with distinct but complementary tasks (see Table 1). Estimates of duration of tasks and their sequence crossed with the availability of politicians for at most 2 full days gave rise to a 5 days schedule, from 11 to 15 June 2007. During the first 3 days, the T-team structured objectives and programmes. In the further 2 days, the DM-team, based on the T-team work, defined and weighted the fundamental objectives, evaluated the programmes and, finally, classified them also taking into account their perceived doability. The two teams met in the first session and on the afternoon of the third day and one technician from each administrative unit also took part in the DM-team’s sessions, assisting their senior decision-makers. Besides the problem-owners, two external experts in the intervention areas of SEDSDH were also present as advisors. Sections 3 and 4 detail the activities developed by the T-team and DM-team, respectively.

Table 1 Tasks of the two work teams
Full size table

To guarantee full concentration of the participants, the decision conferences took place in a retreat setting far from SEDSDH offices. Particular care was devoted to the arrangement of the conferencing environment, motivated by previous experience on its profound influence on the effectiveness of group working (Bana e Costa et al. 2002) (see also Hickling 1990; Phillips and Phillips 1993). The layout of the conferencing room (see Fig. 2) ensured easy eye-to-eye contact between participants and visual access to computer-projected information. Two screens were used to aid the undertaking of the tasks, one of them to display factual data, the other one to show the computer-based models, which were created on-the-spot, integrating objective data and group judgements. The facilitation was guided by the ten principles of process consultation explained by Schein, in particular principle 5 (Schein 1999): “It is the client who owns the problem and the solution” (p. 20). Thus, the process consultants led the teams in “how to think about the issues, not what to think, which is the responsibility of the participants” (Phillips and Bana e Costa 2007, p. 55).

Fig. 2
figure 2

Layout of the decision conferencing room

Full size image

3 Structuring Objectives and Programmes

The structuring phase began with a discussion about the main challenges and concerns of the SEDSDH, which enabled the T-team to reach a shared understanding about the SEDSDH mission. Afterwards, a “Post-it session” (Belton and Stewart 2002; Bryson et al. 2004) was held, developed from the core strategic question “where do you want Pernambuco to be in 2011?”. First, each participant wrote down up to five ideas on Post-it notes, one idea per note, which they believed underpinned the mission of SEDSDH. Next they stuck up their Post-it notes on the wall, trying to place them next to other notes that contained similar ideas; a few new ideas emerged from reading others’ notes and duplications were eliminated, which reduced the initial 60 notes to around 40 that were clustered by similar topics. Then, the meaning and pertinence of each idea were clarified and discussed by the T-team and links with other ideas were identified. Ideas considered relevant and relationships between them were inserted into a Decision Explorer model, gradually giving rise to the group causal map shown in Fig. 3. Discussion around the map enabled successive simplifications and by the end of the second day the T-team had identified the main ends-objectives and means-objectives and their relationships, shown in the simplified map depicted in Fig. 4.

Fig. 3
figure 3

The group causal map

Full size image
Fig. 4
figure 4

The simplified map (ends-objectives are in bold)

Full size image

Meanwhile, for part of the third day of work, the T-team was engaged in an action-generation session on “how to improve the current situation of Pernambuco?”. Actions suggested by the various entities of SEDSDH and other actions that emerged from the “tails” of the causal map (Fig. 3) were clustered into 40 intervention programmes. The technicians were instructed with basic concepts of AIDA—the Analysis of Interconnected Decision Areas included in the Strategic Choice approach (Friend and Hickling 2005) (see also, Friend 2001; Hickling 2001)—which helped them to analyse interconnections between actions, systematise major compatibilities and incompatibilities between actions and take them into consideration in clustering the actions into coherent intervention programmes. Finally, the T-team organised factual information about how the programmes would contribute to improve the Pernambuco current situation in a given end-objective. The most important distinction was between a direct and an indirect contribution. As a matter of fact, one of the “requirements for the constitution of a programme” was its classification according to a typology that distinguishes “finalistic” from “support” programmes: “A programme is considered finalistic if it contains actions that result in goods or services supplied directly to society” (Governo do Estado de Pernambuco 2007, p. 10). For example, programmes 3.1–3.4 in Table 2 are typical support programmes. In the late afternoon of the third day a joint meeting of the two teams was held. The technicians presented the politicians with the causal maps and the programmes and respective contributions that resulted from their structuring sessions.

Table 2 The integrated programmes
Full size table

The fourth day started by the visual analysis of the simplified map in Fig. 4. Assisted by their internal and external advisors, the politicians discussed this “means-ends objectives network” (Keeney 1992) to validate those objectives classified as ends by the technicians. In fact “it may be that what initially appear to be objectives or goals in fact end up being issues (or opportunities or threats) that are means for achieving (or not) a particular goal or objective. Recognizing that something is an issue rather than a goal can help clarify which goals the issue actually affects, along with the determining action needed to effect the goal” (Bryson et al. 2004, p. 84). The discussion gave rise to an agreement on three fundamental objectives: “To promote the social inclusion and protection of people and families” (O1), “to promote human rights, making sure they are universal and guaranteed” (O2), and “to socialise adolescents in conflict with the law and re-socialise the prison population” (O3). Initially, “to improve the management of SEDSDH” was also retained as a fundamental objective of the PPA. However, later on, on the dynamics of the process, it would be considered by the DM-team as part of the organization objective of carrying out the SEDSDH mission, not a fundamental objective contributing to it (Keeney 2006). Next, the DM-team reviewed the programmes and identified inter-sector programmes that could generate synergies between services and a more efficient and effective management. At the end of the structuring phase, at lunchtime on the fourth day, the 40 initial programmes were consolidated into 25 integrated programmes (see Table 2).

4 The Evaluation Phase

4.1 Evaluating the Programmes on Each Fundamental Objective

In an intensively participated decision conferencing session, held in the afternoon of the fourth day, politicians defined and weighted the fundamental objectives and evaluated the contribution of the programmes to achieve them, using MACBETH. Founded on multiattribute value theory (von Winterfeldt and Edwards 1986), the additive value model constructed with the DM-team was then explored to appraise the overall benefits of the programmes and rank them. The method promoted the generation of a compromise within the DM-team through a process of convergent thinking (as defined by Kaner et al. 2007), a key feature for group support (Vogel and Coombes 2010). Following the sequence “present and discuss arguments, vote, then discuss the disparities on the votes if they occur, and then vote again”, initial individual judgements gave rise to group compromise judgements. In typical applications of MACBETH (see, for example, Bana e Costa and Chagas 2004; Bana e et al. 2008, 2012a) judgement elicitation is done with the M-MACBETH DSS. In the SEDSDH’s case, to ease the interaction, a complementary visual aid display was used—the display specifically used for scoring programmes on objective O1 (“To promote the social inclusion and protection of people and families”) is shown in Fig. 5. After an initial brainstorm, each of the nine participants was invited to give a qualitative judgement about the perceived contribution of each of the 25 programmes towards improving the current situation of Pernambuco on objective O1. Whenever a contribution of a programme was not null, each one had to judge its strength, giving an individual “vote” by selecting one of the following MACBETH qualitative categories (Bana e Costa et al. 2011): very weak, weak, moderate, strong, very strong, or extreme contribution. It is worthwhile noting that such a vote corresponds to a judgement of the difference in attractiveness between the consequences of implementing the programme and doing nothing to change the status quo (SQ). The nine individual votes received by each programme are registered in the central columns of the visual display (depicted in Fig. 5), enabling an immediate appraisal of their disparities. These were then subject to group discussion, giving rise to a better understanding of the merits of the programmes and to the eventual revision of individual judgements. This discussion converged to a compromise on group judgements, which are presented already ordered in the column labelled “Difference to Status Quo” in Fig. 5. For example, programme 1.3 (“Social productive insertion”)—aiming to promote the social productive inclusion of the large number of Pernambucans living in extreme poverty—obtained three votes on “very strong” contribution and six votes on “extreme” contribution (see first row of the table in Fig. 5) that converged to “very strong to extreme”. The majority rule was adopted in more controversial cases. Then the differences of attractiveness between consecutive programmes in the group ranking were elicited by a similar voting, registered in the column labelled “Difference to next” in Fig. 5. For example, the “strong” in the first row indicates a strong difference between the two best programmes (1.3 and 6.2) in objective O1. Other MACBETH judgements (not registered in Fig. 5) between programmes were formulated for consistency testing (Bana e Costa et al. 2012b), for example, about differences between the best programme and other programmes. The visual support of the MACBETH group voting procedure was first used and successfully tested in 15 decision conferences involving each one about 10 participants, held in San Juan to appraise long term strategies for Puerto Rico (Puerto Rico 2025 2004).

Fig. 5
figure 5

Evaluation of the contribution of the 25 integrated programmes to objective O1: “To promote the social inclusion and protection of people and families” (the programmes 6.1–8.1 in the group ranking were considered to provide typical good contributions). A similar process was followed for objectives O2 and O3

Full size image

Once all the final voting was carried out (as depicted in the last two columns of Fig. 5, for objective O1), the set of all group judgments were inputted into M-MACBETH (Bana e Costa et al. 2005)—the DSS that supports the application of the MACBETH approach. A score of 100 was arbitrarily assigned to programmes with good contributions (and, of course, zero to null contributions) to improve the SQ in each objective. Note that this corresponds to making explicit and fixing what is meant by a good added value on each objective—not an intrinsically good level of performance or reference targets (as done, for instance, in Bana e Costa and Vansnick 1999 or in Bana e Costa et al. 2012a, respectively). M-MACBETH generated quantitative value scores for the programmes that reconcile all judgments (these scores are generated by linear programming—a detailed and intuitive account of the MACBETH algorithm is available in Bana e Costa et al. 2012b). The DM-team was afterwards asked to analyse the realism of the value scores, and adjust the differences in scores if need be, so that they measure the relative attractiveness of the programmes with regard to objective O1. The scoring process was then repeated for objectives O2 and O3. Figure 6 shows the final value scores, validated by the DM-team, measuring the contributions of the programmes to improve the SQ in the three fundamental objectives.

Fig. 6
figure 6

Scores of the programmes in the objectives: (a) O1—to promote the social inclusion and protection of people and families; (b) O2—to promote human rights, making sure they are universal and guaranteed; (c) O3—to socialise adolescents in conflict with the law and re-socialise the prison population

Full size image

An analysis of the contribution profiles of the programmes was then carried out by the DM-team. For example, it was observed that programme 1.1 (“Pernambuco caring for the street population”) offers a good contribution simultaneously in the three objectives, and programmes 1.3, 1.5, 4.1 and (1+8).1, each one dominating 1.1, are surely at the top of a ranking of the programmes by overall benefit, whatever the relative importance of the objectives. This result was in accordance with the intuitions of the decision-makers. On the other hand, programmes 3.1–3.4 should be at the bottom of the ranking, given their low scores, a conclusion that was not shared by the Executive Secretary of SECOGE, promoter of these programmes (see discussion in Sect. 5.1). However, to produce a complete ranking and identify the most beneficial programmes, the contribution scores in the three objectives should be commensurate and this called for the weighting of the objectives.

4.2 Weighting the Fundamental Objectives

The MACBETH weighting process took place late in the afternoon of the fourth day. Three hypothetical programmes (labelled [O1], [O2] and [O3]) were presented by the facilitator to the DM-team, each one assumed to give a good contribution, similar to the one of programme 1.1, to improve the SQ in one fundamental objective and no contribution in the others:

  •  [O1]—A good contribution “to promote the social inclusion and protection of people and families” and no contributions in O2 and O3.

  •  [O2]—A good contribution “to promote human rights, making sure they are universal and guaranteed” and no contributions in O1 and O3.

  •  [O3]—A good contribution “to socialise adolescents in conflict with the law and re-socialise the prison population” and no contributions in O1 and O2.

Each of the nine members of the DM-team judged the overall attractiveness of [O1], [O2], and [O3] by giving individual MACBETH votes, as displayed in Table 3. This is equivalent to “qualitative swing weighting” (Bana e Costa et al. 2012b), consisting in judging the “importance” of a good improvement to the SQ (i.e. a good swing) in one objective at a time. Note that all individual judgments were at least set as “strong”, an evidence of shared commitment to achieving all objectives. The subsequent discussion led to the compromise judgements shown in the last row of Table 3. [O1] was unanimously considered the most beneficial of the three hypothetical programmes, and [O2] and [O3] were at the end agreed as indifferent. Next, the DM-team was invited to judge the difference between [O1] and [O2], which was taken as moderate.

Table 3 The DM-team’s weighting judgments
Full size table

These judgments were used to populate the weighting matrix in Fig. 7, from which M-MACBETH proposed relative weights for the objectives, which were discussed and adjusted by the DM-team. It is interesting to note that the participant that had voted “extreme” for [O1] argued in favour of taking objective O1 “as important as the other two together”, implying a weight of 50 % to objective O1. This is outside the interval, highlighted by M-MACBETH in the bar chart in Fig. 7, within which the weight of O1 can vary keeping the judgements unchanged. After discussion, the group decided for the final weights of 46 % for O1 and 27 % for the other two objectives.

Fig. 7
figure 7

Weighting process using the M-MACBETH DSS: value tree with the SEDSDH’s objectives, matrix of weighting judgements, and set of weights validated by the DM-team. “No”, “strg-vstrg” and “vstrg-extr” are abbreviations of, respectively, “no difference”, “strong or very strong” and “very strong or extreme” difference in attractiveness

Full size image

4.3 Overall Benefits and Ranking of the Programmes

Table 4 summarises the outputs of the MACBETH additive value model constructed with the DM-team, in which the programmes were ranked by decreasing order of overall benefits, obtained by weighted summation of the respective contribution scores in the three objectives. In the last decision conferencing day, the participants started by confronting the outputs of the model with their intuitions. Extensive sensitivity analyses showed that many individual judgemental disagreements made no significant difference to the resulting most beneficial programmes, namely the nine programmes with good or better than good overall benefits (in italics in Table 4). Occasionally, participants did not agree with this classification because, they said, “some programmes are not realistic given that they are difficult to implement”. This misunderstanding between benefit value and difficulty of implementation gave to the facilitator an excellent opportunity to introduce the concept of “doability” and motivate the group to engage into the next task.

Table 4 Scores of the programmes and weights of the fundamental objectives
Full size table

5 The Prioritisation Phase

5.1 Benefit Versus Doability Analysis

The prioritisation needed to consider not only the benefits of programmes, but also their perceived “doability”, that is, the extent to which the group envisaged, or not, significant difficulties or obstacles to the implementation of each programme. The greater these difficulties or obstacles, whatever their nature might be (political, technical, financial, administrative, logistical, legal, etc), the higher the effort required to implement a programme is expected to be and, consequently, the lower its perceived doability tends to be. This is in line with the concept of “implementability” proposed in other contexts (see for example, Iivari and Ervasti 1994; Shiffman et al. 2005). Figure 8 synthesises the MACBETH process followed by the DM-team to assess the doability of the programmes, based on judgements from “no” to “extreme” doability, from which the doability scores were derived with M-MACBETH, in a 0–100 scale, respectively. As before, the results were discussed, compared with intuitions, adjusted when desired and finally validated by the DM-team.

Fig. 8
figure 8

Doability assessment of the programmes

Full size image

Finally, the DM-team, together with the internal and external advisors, debated the tensions between overall benefit and doability, with the visual support of the \(2\times 2\) strategic graph shown in Fig. 9. The programmes were classified into four categories according to their location in the graph quadrants (Matheson and Matheson 1998):

Fig. 9
figure 9

Benefit versus doability graph

Full size image

“Pearls” (in quadrant II)—very beneficial programmes that have high doability. These are certainly the best candidates to integrate into the 2008–2011 PPA.

“Oysters” (in quadrant I)—programmes of high benefit but difficult to implement. If it was possible to take measures to eliminate the main blockages to its implementation, an “oyster” programme can turn out to be a “pearl” programme.

“Bread and butter’s” (in quadrant IV)—programmes with low benefits but which can be easily implemented; they can be important complements to other programmes and are interesting to produce some short-term results (on this point see the discussion on “quick wins” in Eden and Ackermann 1998).

“White elephants” (in quadrant III)—programmes of low doability and low benefit. They require effort that could be alternatively allocated to implement the “oysters”.

The DM-team particularly appreciated this classification of the programmes and was satisfied with a simple prioritisation: pearls ranked first, then oysters and bread and butter’s. However, the DM-team agreed with the arguments of the Executive Secretary of SECOGE against classifying programmes 3.1–3.4 in this category, for “they are crucial for the success of the SEDSDH mission”, he said. Therefore, the “support” nature of these four programmes could eventually justify their selection. At the end, an alignment with a way forward was reached: at least all pearls and oysters should be selected. A period of reflection before making a final recommendation on the portfolio of programmes to be included in the PPA was agreed. Indeed, “some time is usually needed to digest fully the results, to prepare briefings of recommendations to the decision makers, and to explore the effects of new information that might alter the results in crucial areas” (Phillips and Bana e Costa 2007, p. 66). The SEDSDH’s Head considered the objectives of the process fully achieved and thanked all the participants and consultants before closing the meeting.

5.2 Robustness of Programme Prioritisation

An analysis of the robustness of the programmes’ classification and prioritisation was performed after the decision conferencing process. An effort score was defined as 100 minus the doability score. In Table 5 the programmes are already prioritised by their value-for-effort, defined as the ratio between the benefit and the effort scores. As can be observed, to select all pearls and oysters, as desired by the DM-team, and at the same time respect the prioritisation, the five most efficient bread and butter’s (but no white elephants) should also be selected. The resulting portfolio, constituted by the first 14 programmes in Table 5, provides almost 70 % of the total cumulative benefit, but demanding a cumulative effort of around 50 % of the total effort that would be required to implement all programmes. Using the PROBE DSS (Lourenço et al. 2012), it was found out that, coincidently, the same 14 programmes constitute the best portfolio (i.e. the portfolio providing the highest cumulative benefit) for a limit of 50 % of cumulative effort. Given that the weights of the objectives were agreed after some controversy during the weighting decision conferencing (see Sect. 4.2), it is pertinent to analyse the extent to which a change in the set of weights would not affect the best portfolio. The robustness analysis with PROBE revealed that it would be necessary to change the weight of objective O1 by at least \(-\)10 or +10% (with the corresponding changes in O2 and O3) to alter the composition of the best portfolio. However, this variation of the weights is far from being compatible with the qualitative judgements expressed by the DM-team (see Fig. 7).

Table 5 Programmes prioritised by decreasing value-for-effort
Full size table

6 Discussion

Developing a tool to assist public strategic planning at SEDSDH confronted us with methodological and practice related difficulties and challenges. The most time consuming activity in the socio-technical process described in this paper was to achieve an agreement on the fundamental objectives driving the selection of the programmes to be proposed by SEDSDH as part of the 2008–2011 Multi-Annual Plan of Pernambuco. This is not at all surprising in light of the great diversity of points of view and concerns defended by the representatives of the different entities that constitute SEDSDH. To get them aligned with agreed fundamental objectives and intervention programmes to achieve these objectives was, indeed, a significant achievement of the process. The crucial methodological shift was from an approach centred on discussing alternatives before objectives to a value-focused thinking approach (Keeney 1992), in which the separation of means from ends in the causal mapping sessions was of utmost importance. Once the fundamental objectives were defined, that is the “why” of the SEDSDH’s strategy, the generation of coherent intervention programmes, the “what to do”, was eased. Indeed, we agree with Phillips (2011) that strategy is “about what and why”.

Different modelling paths could have been adopted to evaluate the programmes, all having pros and cons (see Bana e Costa et al. 2008). Consistent with our experience, direct paired comparisons of programmes can help generate a sense of common purpose within a group, in particular when applied in a decision conferencing setting, both in scoring the programmes and weighting the objectives. The same applies to the assessment of the doability of programmes, that enabled to account in the model for the multiple tangible and intangible aspects related to programme implementation and introduced another dimension into the analysis. The “benefit versus doability” graph proved to be a simple yet pragmatic tool to classify the programmes. It may be interesting to observe the structural analogy between the value-for-money and value-for-effort concepts. Note that benefit and effort were both measured in ratio scales with fixed zeros equal to null added-value to the status quo and null implementation effort, respectively. Independently of the technique used for cardinal value measurement, if the value scale built is an interval numerical scale, then, by definition, the scale of differences in value (or added-value) is a ratio scale. This is the case of the benefit scale built (the numerator of the value-for-effort ratio) because it numerically measures added-value to (i.e., contribution to improve) the status quo. So, the fixed numerical score of 0 contribution must be assigned to the status quo and this is guaranteed by including a do nothing option (SQ) in the MACBETH matrix of judgments and fix the score of SQ equal to 0 (see Fig. 6). The same applies to the effort scale, obviously because the effort of doing nothing is null.

Regarding participation issues, although a new way of deciding was adopted at SEDSDH, within a time demanding working-program, initial concerns in that SEDSDH’s staff could not reach a compromise or feel reluctant in accepting the methods in use or the outcomes from the evaluation model were not justified. At the end of the decision conferencing process, the participants had developed a sense of ownership of the final model, a sense of common purpose, as well as a mutual commitment to action (Phillips and Phillips 1993). The participation of the political and technical key players and of external advisors was determinant for the success of the group dynamic. The MACBETH voting procedure aided the group process to converge into a final agreement, promoting the acceptability of the outcome, as the Head of SEDSDH envisaged. In fact, we agree with Mintzberg (1994) that “managers with a committing style engage people in a journey. They lead in such a way that everyone on the journey helps shape its course. As a result, enthusiasm inevitably builds along the way. (...) strategies take on value only as committed people infuse them with energy.” (p. 109)

When invited to revisit the case in a recent interview, three participants—the Head and two top managers of SEDSDH at that time—reported that the participative process had enabled higher integration between the several entities within SEDSDH. This was because a balance was reached between the social and technical components of the process, with participants’ interaction developing within a clear and constructive evaluation system. This balance was successful in combining knowledge, transparency and communication, which are, following Alexander (2006), “challenges that many evaluations in planning fail to meet, even when their methods and procedures represent the best practices we know” (pp. 272–273). Despite its complexity, interactive evaluation led to a reduction in the differences encountered beforehand among the participants’ views and judgements. These results are in line with previous studies highlighting the role of “removing common communication barriers, providing techniques for structuring decisions and systematically directing pattern, timing and content of the discussion” in group decision making (DeSanctis and Gallupe 1987, p. 598). The level of model complexity did not affect communication and understanding by the group, with the multi-methodological approach contributing to an inclusive learning process that promoted legitimacy of the outputs and a sustained justification of the selection made. It was therefore not at all surprising to hear from the three participants interviewed that, at the end, the strategic objectives and programmes defined in the workshop, evaluated in terms of benefit and doability, determined the set of programmes and actions that was inserted in the 2008–2011 PPA, and were agreed by the Governor of Pernambuco to have the highest priority for SEDSDH. Programmes were implemented by the responsible administrative units, to different extents. The interviewees also emphasised that the contact with an innovative decision support method made them better equipped as public managers. It can be inferred that the outcome of the SEDSDH’s decision conferencing sessions was quite successful in terms of, using Lewis terminology (Lewis 2010), meeting effectiveness, meeting efficiency, user satisfaction and quality of decision, with quality of the decision viewed as the success of the decision support process in effectively providing satisfactory “elements of responses to questions” (Roy 2005, p. 5) that motivated the intervention and drove the design of the process (see Sect. 2.1). In spite of this, it should be noted that the multicriteria evaluation model constructed did not explicitly associate descriptors of impacts (“attributes”, Keeney 1992) of the programmes on the objectives. At the eyes of one of the executive managers, “this was not a methodological gap, it just was not included in the working-program because of time and other resource constraints”. But, a consequence was the impracticality of formally analysing and modelling preference dependences between attributes. An alternative pragmatic modelling path would consist in using a constructed descriptor of impacts common to all objectives and with some levels accounting for synergies, as done in Bana e Costa et al. (2002). One should however recognise that this is only a way of bypassing the dependence issue, not of facing it explicitly. A promising line of research in this direction is to explore a feasible way of integrating in a multiattribute modelling framework a complementary tool to capture interdependencies between attributes, compatible with MACBETH—for instance, learning from recent developments in the area of performance measurement of industrial systems (Clivillé et al. 2007). It is also worthwhile to note that the programme prioritisation and selection approach presented in Sect. 5.2 assumed as working hypothesis the validity of the independence conditions (Golabi et al. 1981) common in portfolio decision analysis (Kirkwood 1997; Kleinmuntz 2007; Phillips and Bana e Costa 2007; Salo et al. 2011). The relevant discussion is whether the individual contribution of programmes to improve the status quo can be simply summed up. This is theoretically significant, for programme benefits are measured in a scale with a fixed zero (Clemen and Smith 2009), and it is also substantively significant (French 1986) in the specific context of the SEDSDH, for taking the programmes as value independent sounded reasonable to the SEDSDH’s decision-makers. However, programme additivity may not always hold, but the PROBE DSS is now able to model programmes’ interactions (Lourenço et al. 2012) within the methodological frame described in this paper. Finally, one should research the extent to which the MACBETH questioning-answering mode for qualitative value assessment is prone to behavioural biases (Fasolo et al. 2011).