Introduction: The Growing Importance of Cross-Sector Partnerships

Cross-sector partnerships are one of the most exciting and dynamic areas of research and practice within business and society relations. Partnerships that bridge different sectors (public, private, and nonprofit) are thriving around the world. Thousands of cross-sector partnerships are currently active and/or under consideration or development, and there has consequently been a dramatic increase in the management and policy literature on cross-sector partnerships (Gray and Stites 2013; Branzei and Le Ber 2014). Austin (2000) was the first to label these alliances the collaborative paradigm of the twenty-first century (Van Tulder 2010).

The central aim of many cross-sector partnerships is to solve economic, social, and environmental problems through collaboration (Crane 1998), often by addressing institutional and regulatory voids (Fransen and Kolk 2007) by providing social goods such as clean water, health, or education (Warner and Sullivan 2004). Hence, cross-sector partnerships typically emphasize an ‘imperative to realize benefits for the wider community rather than for special interests’ (Skelcher and Sullivan 2002, p. 752). Partnerships generally address the social responsibilities of participating organizations, either in response to external pressures (reactively), in anticipation of potential social issues that may arise in the future (proactively), or as part of the process of interaction by adapting to emergent issues (adaptively) (Seitanidi 2008; Van Tulder et al. 2014). Cross-sector partnerships are, therefore, expected to deliver improved and innovative solutions for economic, social, and environmental problems via the combination of the capacities and resources of organizational actors across different sectors (Brinkerhoff 2002a, b; Gray 1989; Huxham and Vangen 1996).

The idea that cross-sector partnerships are a new paradigm for strategy across the different sectors is manifested in their growing empirical pervasiveness. Large companies have come to appreciate the potential for cross-sector partnerships to contribute to long-term competitive advantage. Early evidence suggested that the one hundred largest firms in the world were on average involved in about eighteen cross-sector partnerships with ‘non-market’ actors (PrC 2010). In addition, governments have seen cross-sector partnerships as innovative ways of producing public goods in collaboration with firms (Clarke and Fuller 2010) and NGOs (Brinkerhoff and Brinkerhoff 2011). Since the early 2000s, international organizations such as the United Nations and the World Bank have embraced public–private partnerships (PPPs) as a means of providing global public goods like environmental protection or poverty alleviation (Glasbergen et al. 2007; Rivera-Santos et al. 2012). While governments have traditionally used PPPs to build-up ‘hard’ infrastructure such as roads and water works, they are now increasingly experimenting with using PPPs for ‘soft’ issues with varying constituents and aims (Dixon et al. 2004; Milliman and Grosskopf 2004; Skelcher and Sullivan 2002; Teegen and Doh 2003). Finally, cross-sector partnerships are increasingly being adopted by many civil society organizations in preference to a confrontational approach toward firms and governments in order to develop novel solutions to old problems, thereby aiming to increase the efficiency and effectiveness of their activities (Le Ber and Branzei 2009; Galaskiewicz and Colman 2006; Hamann et al. 2008; Jamali and Keshishian 2009; van Huijstee and Glasbergen 2010; Laasonen et al. 2012; Seitanidi and Crane 2014; PrC 2011).

With this exponential growth in activity, the question facing many actors in society has shifted from one of whether partnerships with actors from other sectors of society are relevant, to one of how they should be formed organized, governed, intensified, and/or extended. Arguably, assessments of the efficiency and effectiveness of partnerships in addressing their intended goals are the most critical elements in partnership decisions. Many early partnerships were characterized by an absence of formal planning (Austin 2000; Jamali and Keshishian 2009; Seitanidi et al. 2010), and modest or partial consideration and evaluation of anticipated outcomes and impacts (Margolis and Walsh 2003). The anticipated benefits for the actors involved in cross-sector partnerships have been extensively discussed in the literature, but realized outcomes, benefits, and impacts are much less often discussed even in the older form of public sector partnerships (Provan and Milward 2001; Leach et al. 2002; Arya and Lin 2007) indicating the challenges that exist in monitoring, reporting, and evaluation in practice as well as in applying or developing appropriate methodologies in research.

Cross-sector partnership research is characterized by widely dispersed and multi-disciplinary theoretical roots (cf. Gray and Wood 1991; Gray and Stites 2013; Hull et al. 2011) as is the case with its methodological approaches employing a multitude and mixture of methods, which has resulted in a toolkit that has ‘grown large and heavy to carry’ (Branzei and Le Ber 2014, p. 231). Researchers switch from one area to another, whereas words, concepts, and definitions are embraced with sometimes limited reference to each other. Hence, although there is a growing abundance in diversity, there is a lack of focus and co-ordination of methods (Crane and Seitanidi 2014). Researchers have largely tried to complement each other, rather than entering into a productive conversation regarding significant points of theoretical or methodological disagreement. This is a typical sign of a field in a build-up phase, in which the diversity of approaches can lead to productive development of the field. In addition, the booming attention to the issue of partnerships creates considerable demand for rapid scans and practical insights, with often limited space and scope for fundamental reflection and consolidation of knowledge. Moreover, methodological diversity also creates transaction costs that can hamper progress in a later phase and can also lead to the persistence of superficial or ideological discussions.

It is our contention that there is an urgent need for cross-sector partnership research to pay greater attention to the monitoring, reporting, and evaluation of the outcomes and impacts on social problems of partnerships. This is necessary to inform and support the legitimacy and credibility of partnerships as an effective and efficient approach to solving complex social and environmental issues, as well as in determining their necessary limits. Importantly, enhancing the impact of cross-sector partnerships requires greater attention to developing shared understanding about the meaning of impact in partnerships. Extant literature has examined what social partnerships are about (the “what” question), the motives and drivers behind such collaborations (“why” questions), and the process of forming and implementing partnerships (“how” questions). Although research about the outcomes of partnership is limited, research on the impact of partnerships i.e., looking whether partnerships make a difference to society (“so what” questions) is mainly grounded on anecdotal evidence employing prescriptive and “best-practice” reasoning. There is a lack of convincing evidence based on monitoring, reporting, and evaluation. Despite these challenges the proximity, almost in real time (Branzei and Le Ber 2014), between partnership research and praxis holds high potential for the development of relevant and useful theory for practice (Seitanidi 2014) as well as methodologies to addresses the challenges described above.

In this paper, we provide a first step towards initiating, organizing, and developing a productive exchange between research on cross-sector partnerships and impact assessment. The paper begins by discussing the growing need for impact assessment in cross-sector partnerships (“The Growing Need for Evidence-Based Impact Assessments” section), taking stock of the latest insights and discourses in two relevant areas: the cross-sector partnership and the impact assessment literatures (“Impact Assessment Challenges” section). We then develop a framework to guide future research in partnership effectiveness and efficiency (“Framing Partnership Impact Assessments: Two Complementary Roads” section). We thereby distinguish four basic impact pathways or loops of partnerships that create four different “orders” of impact. Each adds a different lens through which to systematically examine the different types of partnership impacts. This framework is intended to enable a more productive exchange of knowledge in future research across both areas. In particular, we more precisely categorize impacts arising from partnerships in order to help facilitate the selection of appropriate methodologies for impact assessment. Finally, we frame the four papers of this special issue along the various impact orders (“Impact Orders in the Special Issue” section) as a way of illustrating the usefulness of the framework and positioning the papers in terms of their contribution to the debate on enhancing the impact of cross-sector partnerships.

The Growing Need for Evidence-Based Impact Assessments

Despite their growing popularity, precisely evaluating the value added of partnerships has proven difficult, partly because of the dynamic and evolving nature of cross-sector partnerships. While recent research developments are beginning to address this issue, the lack of attention to impact assessment within partnership research was originally strongly influenced by the relative novelty of cross-sector partnerships, their diversity, the lack of available resources, limited research interest, and the lack of appropriate methodologies. The inherent complexity and diversity of cross-sector partnerships presents a number of analytical and methodological difficulties in assessing the impact of partnerships as they often require sophisticated methodologies, multi-level tools, and longitudinal research designs that are not easy to develop, implement and elaborate. A central issue here is the so-called attribution problem (Brinkerhoff 2002a, b): namely, the problem associated with isolating the impacts of a specific cross-sector partnership from other confounding contributing influences. The more complex the issue the partnership is intended to address, the more difficult the attribution problem becomes. Therefore, despite a dramatic increase in the management and policy literature on cross-sector partnerships, the field faces a number of pressures to develop better ways of thinking about and assessing impact.

Organizational Pressure

One set of pressures toward greater attention to impact has come from participating organizations themselves. To begin with, an absence of proven impact can affect the legitimacy of organizations investing time and money in partnerships, in particular when the stated ambitions are high. Many organizations place high hopes on partnerships to solve some of the problems they face due to market, civic, and governance failures (Kolk et al. 2008) or in support of extending their strategies into new areas. There exists the danger of taking credit for results that the partners cannot achieve (Ebrahim and Rangan 2013). In general, the pressure on organizations to measure performance and establish “what works” also in more complex areas like social programs, has increased (Epstein and Klerman 2013; Khagram and Thomas 2010; White 2009). Therefore, there is a greater emphasis on the consequences of partnerships (Biermann et al. 2007) or impact instead of the more traditional focus on inputs and output effects. This is also accompanied by increases in budgets for impact assessment and stepped-up monitoring requirements in international development initiatives (Liket and Maas 2012). For example, a survey among NGOs and firms in the UK (C&E 2013) showed that companies, and to a lesser extent NGOs, consider it vital to “prove” not only societal considerations within their business practices, but also the impact of their activities. For all major societal actors, clearly demonstrating what impacts have arisen from partnerships is becoming more important.

Although nonprofit organizations have a longer tradition in social impact assessment due to their need to document making a difference to the social issues they tackle to a wider range of publics (Mulgan 2010), the push for concrete impact assessment at the moment seems acute also among companies, as they are interested in cost/benefit assessments. Company-induced partnerships tend to address less complex problems which can be more susceptible to systematic evaluation. Business involvement in more complex partnerships registered by the UN or in climate change is relatively limited (Pinkse and Kolk 2012), and thus there exists a relatively straightforward push for impact assessments and social performance metrics by corporations. In particular, in the area of CSR strategies, the demand for impact assessment has increased to enable reporting, prevent allegations of window dressing, and to legitimize the societal involvement of organizations. This tendency has created a competitive “market” for impact assessment. Although a wide range of impact assessment models are available in the private (Liket and Maas 2012) and nonprofit (Maas 2009) sectors, fertilization of impact assessment models across sectors remains relatively limited.

In most of the extant impact assessment frameworks, partnerships have not yet systematically been taken into consideration and, reflecting this, there is very little empirical evaluation of the potential of partnerships to contribute through attribution to specific impacts. However, the higher political stakes involved in partnerships makes the assessment technique itself potentially contentious. The measurement of the impact of PPPs, for instance, has been particularly difficult because of a lack of baseline metrics, and an unwillingness by participating managers to disclose the impact effects on their own organizations (Maas 2012). The reasons for this are related to measurement problems, but also to the general feeling that it is more important to start participating in a partnership than to actually question or measure the exact starting position of each participant too much. In addition to measurement difficulties, including the nonquantifiable value of partnerships, the temporal dimension and the multi-causality of partnerships (Austin et al. 2006) add to the impact assessment challenges. For example, when cross-sector partnership brokers are initiating a partnership they face a trade-off between seizing the opportunity to start a partnership as a coalition of the willing and the desire to assess in more detail the exact nature of the problem and the motivations of the potential partners (Stadtler and Probst 2012; Wood 2012) which would require significant time and effort to establish the partnership’s base line. However, a hampered assessment of the starting position of a partnership affects its dynamics as well as the ability of the participants to keep track of progress, making it difficult to assess impact convincingly and consistently.

Research Pressure

A secondary trigger for impact assessment is the pressure from partnership researchers regarding the legitimacy and effectiveness of partnerships, due to a persistent questioning of whether partnerships are a “panacea” or “hype” for solving social problems (e.g., Barnes and Brown 2011). Gaps in regulation and governance (Rivera-Santos and Rufín 2010) or democracy (Bäckstrand 2006) are not easily, if ever, filled by partnerships. New institutional voids have appeared and partnerships have arguably crowded out other relevant interest groups or introduced “solutions” that are as controversial as the problems they were intended to address (Mert and Chan 2012). Relatively little is known of the contribution of cross-sector partnerships to wider societal goals, such as the millennium development goals (Utting and Zammit 2009). The greater difficulty of doing research into these broader social problems in which attribution problems are most severe has created a lack of empirical findings (Babiak 2009) as well as limited theoretical development. Despite the growth in the scale and scope of partnership research in international development, for example, the field arguably continues to have ‘an impoverished theoretical appeal, which is under-defined, poorly scrutinized, and rather unconvincingly utilized as a guiding concept in applied practice’ (Barnes and Brown 2011, p. 166). Others witness an overuse of the term partnership (Brinkerhoff and Brinkerhoff 2011). United Nations organizations, governments, NGOs, and firms have therefore started to call for better and more evidence-based impact assessment methodologies (Lund-Thomsen 2009).

These circumstances reiterate the importance of moving the discussion on impact beyond generalizations and toward more concrete evidence-based insights. The lack of proper techniques provides no excuse for not engaging in monitoring and evaluation (Austin and Seitanidi 2014). There is a recognized need ‘to fine tune further efforts, and assesses when and under which conditions different types of partnerships do and do not work, and in which cases other mechanisms may be more effective’ (Kolk 2014, p. 37). In impact assessment terms, this emphasizes the importance of understanding the so-called “counterfactual”—the question of what would have happened anyway without the intervention of the partnership—in order to more precisely frame the research on partnership impact.

Impact Assessment Challenges

The discourses in both areas of research that are central to this discussion—partnership research and impact assessment—have largely progressed independently. Recently, some interaction has appeared, but without much cross-fertilization. Nevertheless, partnership researchers are clearly becoming more interested in impact evaluation, while impact assessment researchers are showing more interest in networks and complex constellations of actors. In this section, we explore how and to what extent cross-fertilization can actually be achieved by noting a number of separate developments, common challenges, and possible approaches needed for an effective framing of insights from both areas.Footnote 1

Conceptual and Definitional Challenges

A primary challenge to greater cross-field engagement is definitional. Both areas of research are still (pre)occupied with basic questions such as the definition of ‘cross-sector partnerships’ and ‘social impact.’ Some refer to this as definitional ambiguity (Glendinning 2002) others as definitional “chaos” (Ling 2002; White 2009).

In impact assessment research, for instance, there is still discussion regarding what constitutes the difference between the outputs, outcomes, and impacts of particular actions or programs. A number of authors and institutions make a distinction between immediate, intermediate, and ultimate outcomes. There is growing consensus, however, that “outputs” often refer to immediate effects on the participating organizations, while “outcomes” relate to intermediate direct effects on the targeted communities, and “impacts” to long-term and net effects (direct and indirect effects) on whole issues. Liket and Maas (2012) note that this delineation of an impact chain causes practical problems because long-term effects are difficult to measure, in particular, for more complex problems. Lack of data makes many partnership projects appear to be “impact-less,” in spite of considerable achievements having been made. This problem can be addressed by defining different “orders of impact” which leaves the basic idea of an impact chain of effects intact (White 2009; Ebrahim and Rangan 2010), but nevertheless includes different levels of impact. This is elaborated in more detail in “Framing Partnership Impact Assessments: two Complementary Roads” section.Footnote 2

Another definitional debate arises in cross-sector partnership research, where there is still discussion of classifications and typologies of partnerships within and across “sectors” (Beisheim 2012; Pattberg et al. 2012; Kolk 2014). For instance, much partnership research focuses on a “third” sector that constitutes civil society, or examines the distinctive character of partnerships involving “public” versus “private” sector actors. Gray and Stites (2013) describe a fourth sector—“community”—as distinctive from NGOs. In general, however, NGOs have been considered the representatives of communities, which makes them part of a wider category of civil society organizations. In other studies—and certainly in the policy discourse—knowledge institutes are considered a separate sector, but mostly they are considered as hybrids between public/private and profit/nonprofit sectors.

A similar discussion exists regarding the definition of the notion of a “partnership.” A systematic literature review of the partnership literature over the last twenty years reveals that most contributions refer to “alliances,” or to “collaboration” in general rather than to “cross-sector” partnerships specifically (Drost 2013). The lack of a uniform analytical frame makes it difficult to compare partnerships and evaluate the cost and benefits (Glendinning 2002). One approach to this definitional ambiguity is to provide a more narrow or theoretical definition of partnerships. For instance, it has been suggested that the characteristics of partnerships might be used to more precisely define “partnerships” as arising only in circumstances where partners can be considered equal and the partnership non-hierarchical (Glasbergen et al. 2007), or where partners share a high degree of mutuality, accountability, and transparency (Brinkerhoff 2002a), or in those conditions where partnerships primarily concern “risk” sharing agreements (Ministry of Economic Affairs 2012). While such a choice potentially facilitates impact analysis by making comparison easier, it also limits research to a relatively small subsample of the observed phenomenon.

Methodological and Measurement Challenges

Both areas of research increasingly acknowledge the conceptual and methodological pitfalls seen in extant research. Many of the impact measures developed in one sector, even if they are used by many sectors (such as “social return on investment”), are not suitable for the more complex organizational forms of cross-sector relationships where multiple actors from different sectors interact and co-create impact. Partnerships represent a wide variety of organizational forms, interests and expectations which makes it much more difficult to define “success” than evaluating a single organization (Provan and Milward 2001).

One methodological solution that has been proposed for this problem has been the linking of the outcome of a partnership to the objectives as defined by the participants. This approach addresses only part of the impact challenge, as partners might raise non-compatible and unrealistic expectations, or even define the issue or problem differently to begin with. Evolving expectations, targets, and constituencies makes it exponentially more difficult to research partnerships than to research single organizations (Selsky and Parker 2005; Toulemonde et al. 1998). Moreover, changes in conditions over time can affect partners differently (Vellema et al. 2013).

An important implication of this discussion is that the effectiveness of partnerships is strongly context dependent and needs to be considered in its interaction with context. This interaction can create indirect and unintended effects that affect the overall impacts of partnerships. There is a growing literature that tries to take the context of partnerships into account in order to make a diverse assessment of impact, in particular in contexts characterized by institutional gaps (Kolk and Lenfant 2012; Mair et al. 2012). Many of these impact measures, however, are still based on “perceived impact” rather than objectively defined impacts. Moreover, taking the context of the partnership into account can require that new levels of analysis are introduced, such as the global value chain, which introduces its own methodological challenges. Furthermore, the nature of the institutional gap that the partnership addresses influences its effectiveness: are partnerships primarily aimed at filling gaps as regards regulation, participation, implementation, resources, and learning (Seitanidi and Crane 2009; Kolk 2014; Pattberg et al. 2012), or are they aimed at “creating opportunities” and creating value (Austin and Seitanidi 2014)? Attribution of the impacts of cross-sector partnerships under such complex clouds of intertwined conditions has created legitimate ground for questioning the relevance, effectiveness, and replicability of partnerships (Roche and Roche 1999; DAC 2008).

Partnership research, despite its fragmented nature, has resulted in considerable knowledge on the drivers and motivations of cross-sector partnerships (Gray and Stites 2013), which influence partnership characteristics (Laasonen et al. 2012), process issues, and even some output and outcome characteristics. Austin and Seitanidi (2012a, b, 2014) identified a large number of value drivers and outlined a collaboration process “value chain” and an outcomes assessment framework. Gray and Stites (2013) systematically examined the state of the literature on cross-sector partnerships (for development) and highlighted numerous positive outcomes and a more modest list of negative outcomes for individual stakeholders (firms, NGOs, governments) in the literature. However, they also note the relative absence of the outcomes of partnerships on communities and on the environment and suggest that evidence of multi-sector partnerships’ effectiveness still remains largely anecdotal and prescriptive (ibid: 54). More broadly, the motivations for developing cross-sector partnerships for more complex problems are widely acknowledged. But many of these motivations are strongly related to perceived or anticipated impact, the value creation potential and the ambition to effectively contribute to solving wicked problems. Many of these aims are difficult to measure or are difficult to attribute to the specific partnership and therefore cannot yet be substantiated. The more complex the problems are addressed by the partnership (either directly or indirectly), the more additional research is required.

Some studies have taken a more critical perspective: for instance presenting the effects of PPPs as the outcome of a struggle between a variety of actors (Lund-Thomsen 2009), observing that little is known about their contribution to wider goals (Utting and Zammit 2009), demonstrating that community development partnership initiatives have only limited positive impacts (Idemudia 2009), or noting that companies are not adequately monitoring partnerships to see whether they actually enact their strategic investment (Esteves and Barclay 2011). Critical studies tend to reiterate the importance of context (Rein and Stott 2009) and of taking the consequences for communities into account. Most studies conclude that the impacts of partnerships need to be addressed at three levels of analysis: community, network, and organization (Provan and Milward 2001; Babiak 2009). Recent research also proposes a fourth level of analysis, namely the individuals within participating organizations (Seitanidi 2009; Kolk 2014).

One of the most noticeable developments within the impact assessment literature has been the spread of experimental methods with random assignment to treatment and control groups (Duflo et al. 2006). In a number of high profile areas, these studies adequately addressed the so-called macro–micro paradox of international development aid: how to link the benefits of development projects to impact at the macro-level (Liket 2014). The experimental method is typically considered the most rigorous method currently available, and is particularly effective at providing robust evidence on what works and what does not within less complex partnerships with a limited focus. However, experimental methods have considerable methodological limitations when applied to more complex cross-sector partnerships. For example, the experimental method has difficulty in taking into account spill-over effects from pilot-intervention areas to other areas, which makes the distinction between “treated” and “control” groups difficult (Ravallion 2010). Additionally, the relevance of context (Epstein and Klerman 2013) and the impossibility of establishing random and real control groups (Bamberger et al. 2010) in cross-sector partnerships make the technique less appropriate to many partnership evaluation problems. As a result, quasi-experimental methods and more qualitative indicators are introduced in impact assessment research in which contextual variables are included (Vellema et al. 2013).

A recent systematic review of the available evidence on the sustainable development impact of PPPs gives an illustrative example of the difficulties encountered in impact measurement. Bouman et al. (2013) identified 47 studies that could qualify as “valid” and insightful—taking a broad definition of impact. Eighteen case studies and twenty-nine reviews were included. Studies mainly focused on PPPs in healthcare, infrastructure, water supply and agriculture. The review concludes that the rationale for introducing PPPs as novel way to address sustainability issues is mostly based on resource mobilization motives—due to various forms of failure of each party individually—rather than for effectiveness reasons. Partners’ goals and missions are often defined in a general way, while criteria for measurable, attainable, relevant, and timely objectives are usually absent (ibid: 7). Most of the studies reviewed consider the effects of partnerships on outputs, not on outcomes or impact. This leaves the attribution question (if the effects can be linked to the partnership) largely obscure. The review found only one study with a counterfactual. Attribution of effect to particular PPP features in the complex area of sustainable development therefore does not seem possible using the most robust impact measurement techniques. This is further reiterated by practice.Footnote 3

Practical Challenges

Developments in practice are beginning to encourage and support closer links between partnership and impact assessment research. International organizations such as the OECD and bilateral donor organizations have started to require organizations to come up with a so-called “Theory of Change” (ToC) that explains the intended results (impacts) of proposed partnerships at their outset. This requirement goes beyond a simple logical framework (DCED/OECD 2010). A ToC provides critical reflection on the hypothesized causal relations and underlying assumptions of an intervention strategy that results in a sophisticated theory that explains why an intervention might be expected to generate the intended change (Vogel 2012).Footnote 4 Since we are in the early stages of the large-scale adoption of ToCs by partnership projects, it is difficult to assess their practical relevance at the moment but research on partnership impacts is likely to be facilitated through more explicitly specifying the assumptions of an intervention.

Practitioners, however, are not necessarily pleased by this development. While some argue that higher levels of detail will by definition result in more reliability of the intervention strategy (Michie and Prestwich 2010), others acknowledge that there is a trade-off between detail and the time spent on formulating a ToC (Vogel 2012). Regarding the question of how to develop useful frameworks and ToCs, Valters (2014) points to the risk of relying too much on “scientific evidence” produced in highly controlled settings. The kind of results that can be accumulated in these settings is very different from the complex environments of partnerships. For this reason, Craig prefers that ToCs include causal hypotheses that are based on a rationale other than evidence such as logical or ethical arguments.

A more recent trend in the practice of the ToCs is to no longer speak of “impact” but rather search for “plausible effects.” This approach is also gaining support from impact investors and influential foundations such as the Bill and Melinda Gates Foundation. Such an interpretation of the ToC resonates with the advice given by both Rogers (2009) and Davies (2005) for the most complex types of intervention. Because complexity arises from interdependent agents that influence each other but act according to fairly predictable rules, it is best to adopt a network perspective on change. In practice, this implies that ToCs become less a representation of change in terms of a sequential process and more a ‘short list of simple rules’ (Rogers 2009, p. 43) according to which the system is expected to behave. These authors argue that in complex environments an overly detailed ToC is ‘counter-productive because it stifles creativity and innovation’ (ibid: 44).

Taking Stock

No analytical framework for impact assessment exists yet that is applicable to all partnerships (Babiak 2009; Atkinson 2005; Maas 2012). Taking all the previous challenges into account, we propose that an analytical framework for partnership impact assessment ideally should take a large number of dimensions into account to constructively advance the field:

  • The ultimate ambition of the impact assessment and the role taken by the researchers: does it aim to understand potential impacts or to “prove” the value added of the partnership, or both?

  • The appropriate level of analysis for the assessment: micro, meso, macro, or their interaction (Asthana et al. 2002); is it about the impact of the partnership on individuals, organizations, the partnership, the issue or the community/society?

  • The distinction between “output” and “outcome” and between “immediate,” “intermediate,” and “longer term” outcomes (also referred to as “sustainability”) as a relevant proxy for impact;

  • The nature of the problem that the partnership addresses and the benchmark of success that is therefore required for its impact assessment: “simple” problems require different impact assessments than “wicked problems”;

  • The degree to which affected partners are adequately involved in deciding and assessing impact;

  • The intervention logic as defined in a more or less detailed theory of change; relatedly, how to define a sequence of “plausible effects”;

  • The possibility to specify control or benchmarks groups;

  • The extent to which the partnership context has to be taken into account and at what level (region, network, country, supply chain);

  • How to account for typical partnership effects: spill-over, indirect, and unintended effects;

  • Should the focus be primarily on efficiency or effectiveness of the partnership?

  • What part of the impact chain can be left un-researched (black box) and what does that imply for replicability and generalizability of the assessments?

The next section explains how we propose to address these issues to enable a systematic and constructive approach to impact assessment in cross-sector partnerships.

Framing Partnership Impact Assessments: Two Complementary Roads

In both partnership and impact assessment research, the areas we are concerned with in this paper, we see two traditions developing that largely define the struggle of organizations and researchers to perform meaningful impact assessment (Liket and Maas 2012). This struggle has been discussed as the difference between “evaluators measuring impact” and “impact evaluators” (White 2009).

The first perspective of evaluators measuring impact takes the partnership as of point of departure and defines impacts as the effects at the final level of a causal chain. This view adopts an outcome perspective of partnerships (Austin and Seitanidi 2014), where the level of sophistication depends on the degree to which it is able to include different types of effects during the partnership implementation including positive and negative, direct, and indirect, short-term and long-term, intended and unintended effects that ultimately lead to outcomes. Partnership research on partnership value creation (Bing and Epstein 2013) looks for “plausible effects” where impact evaluation becomes primarily framed as a learning approach that is focused on helping managers and stakeholders to learn more about their interventions and on understanding why and how outcomes and impacts are realized or not (Mayne and Stern 2013; Gray and Stites 2013, p. 8). This approach takes a relatively instrumental perspective of partnerships, by, for example, seeing them as the extension of CSR implementation (Margolis and Walsh 2003; Seitanidi and Crane 2009) and prioritizing the organizational actors’ direct benefits (Seitanidi 2010). However, evaluators adopting the outcome approach to measure impact often do not move beyond a first assessment of output—leaving longer term outcomes and effects open for follow-up studies.

In contrast, the second perspective of impact evaluators takes the (social) issue as the point of departure. This perspective sees as its objective providing evidence that partnerships actually make a difference to the social issue. The strictest application of this perspective follows strong methodological rigor, associated with experimental and quasi-experimental methods and employing randomized control groups. Another consideration in this type of research is the crowding-out effect to non-involved stakeholders. It is not surprising that this line of research is challenging when applied to complex problems and cross-sector partnerships, as the ambition to define control groups that operate under more or less the same circumstances—but without the intervention of the partnership—is exceptionally challenging. However, as this type of impact assessment seems to become quite dominant as a source of research funding, partnership practitioners and researchers might need to consider it in the future in order to provide robust evidence for addressing wicked problems by capturing partnership impact.

Enhancing the impact of partnerships involves addressing multiple measurement problems simultaneously and combining both approaches mentioned above aiming also to address the associated challenges identified in the evaluation literature (Liket and Maas 2012). How these approaches are combined depends on the ambition and available resources to researchers. This paper argues that the state-of-affairs in both areas of research has sufficiently progressed in order for this productive exchange to be realized.

A consecutive argument is that the impact of cross-sector partnerships can best be enhanced by addressing how to define different routes through which partnerships actually create effects/value, how to assess whether these routes are more effective than other possible routes (the counterfactual and effectiveness), define what factors are of influence to the suggested impact chain (the logic) and what kind of research is needed to enhance the efficiency of the chosen partnering approach. The approach we propose in this paper is to search for a common framework in which to document and assess various impact pathways of cross-sector partnerships. The complexity of the exercise in measuring impact will increase with the complexity of issues and partnership configurations. We propose to define the impact order of the partnership as a classification frame to be able to compare and develop different theories and methods in the area of partnership research. By classifying different approaches toward impact assessment, overstatements of particular strands of research can be prevented.

The Partnering Monitoring and Evaluation Framework that we embrace takes the growing practice of sketching impact value chains and the quest for greater attribution and counterfactual into account.Footnote 5 This frame is based on Van Tulder and Maas (2012) and contains two dimensions: (1) an impact value chain that documents the actual steps of the partnership from issue definition through to impact; (2) an effectiveness assessment approach that assesses the fit and value added of the partnership to the actual societal problem. Figure 1 shows the most relevant constituting factors of these two dimensions in an integrated model. Research on partnerships usually zooms in on specific parts of the model, while taking the other parts as given.

Fig. 1
figure 1

The Partnership monitoring and evaluation framework. Source Van Tulder and Maas (2012)

This framework presents a chain of results in which organizational inputs and activities lead to a series of outputs, outcomes, and ultimately to societal impacts (Ebrahim and Rangan 2010). In contrast to activities and outputs, impacts actually capture the effects on society as a result of organizational efforts, instead of measuring intentions or activities undertaken by organizations (Maas 2009). While intentions and outputs are related to the providers of the product, activity or service, outcomes and impacts are associated with beneficiaries (Kolodinsky et al. 2006) and other stakeholders. Impacts include both intended and unintended effects, negative and positive effects, and long-term and short-term effects (Wainwright 2002).

Impact Value Chain

The impact value chain (based on e.g., Wainwright 2002; Maas 2009; Ebrahim and Rangan 2010; Maas and Liket 2011; Austin and Seitanidi 2014) includes the following elements:

  • Issue refers to the definition of the social issue being addressed by the partnership. The first step in achieving any kind of impact is for participants to agree on the articulation of the social issue they are seeking to tackle (Austin and Seitanidi 2014), the responsibilities involved and the roles that can be taken by the partners (Van Tulder and Pfisterer 2014). (Social) issues can be defined either in terms of problems or opportunities.

  • Mission acts as the linking pin between the issue and the input. Where the partnership is problem driven, the partnership can be considered to be more “strategic” and long term, while where the partnership is more solution/opportunity driven, the partnership can be more temporary and tactical: once the ambition of one party has been achieved the partnership can be terminated. The latter can for instance be expected from corporate-NGO partnerships that aim at the creation of markets at the bottom of the pyramid. The same mechanism applies for NGO-corporate philanthropic “partnerships” in which parties are primarily interested in a sponsoring relationship for mutual branding.

  • Inputs are the resources and capabilities (money, staff time, capital assets, and commitment) provided to achieve the partnership’s mission. In cross-sector partnerships at least three types of actors provide distinct types of inputs in varying constellations: public actors (governments), private actors (firms), and club/community actors (civil society). Partnership research that focuses on the formation of partnerships in particular considers this factor (PrC 2012). The success of the partnership relies on the competencies and resources that are brought in by each partner. The resource-based view, network, and stakeholder theories are often applied in this research area.

  • Throughput is the actual dynamism, execution and implementation process of the partnership, sometimes referred to in evaluation studies as “activities” (OECD-DAC, 2011). The throughput dimension focuses on the structure within which partners work towards the partnership objectives, which depends on the (1) number and nature of participants, (2) the roles that are adopted by the participants, (3) the arrangement and degree of internal dependencies chosen, which in turn is influenced by (4) the position of participants as primary or secondary stakeholder in the project (cf. Fransen and Kolk 2007) and the degree to which the partnership is “institutionalised” in the participating organizations (Seitanidi 2010; Van Huijstee and Glasbergen 2010). Partnership research that concentrates on this dimension in particular takes process issues into account, focusing on a variety of factors including governance, accountability, agency, transaction costs, decision-making structures, and power.

  • Outputs are results that a participating organization or project manager can measure or assess directly. Output represents the deliverables or what will be accomplished as a result of the combination of inputs and activities. A first output criterion is the extent to which the individual objectives of each participant have been achieved. Did the partnership fulfill the original objectives of the participants or not, or did it perhaps even add to them? A second output criterion is the extent to which the project objectives have been achieved. Did the partnership result in concrete and tangible results? What are the “benefits” for each of the participants (in terms of, for example, profits, members, legitimacy, exposure, and moral capital)? A final criterion is the extent to which the partnership brought about goal-alignment (Kolk et al. 2008) and as a consequence scale-up or termination of the project. A project might not be sustainable if it remains dependent upon the continued financial support of governments or other partners. So another question might relate to whether the period of engagement of each individual partner has been sufficient to guarantee the sustainability of the project. The majority of empirical partnership studies have concentrated on the output dimension of the impact value chain with sometimes extrapolations to longer term (outcome) effects.

  • Outcomes are the benefits or changes for individuals, communities, or society at large after participating in, or being influenced by, the activities of the organizations and the partnership. Outcomes are, unlike inputs and outputs, much more comprehensive and should be translated to the extent that the goals of all organizations are achieved. Commonly, the organization running the program targets these results but may itself not have the knowledge or expertise to evaluate whether an outcome has been achieved. More critical approaches to partnerships have considered this dimension in particular, and have frequently pointed at the lack of outcomes attributable to partnerships.

  • Impacts are the ultimate changes that one effects through the partnership. It addresses positive and negative, short-term and in particular long-term effects produced by the partnership, directly or indirectly, intended or unintended. The impact of the partnership can be measured at the level of the partners, the stakeholders and the system.Footnote 6

Efficiency/Effectiveness Assessment

The efficiency dimension of a partnership can be seen as the internal value-added of the partnership, which may be assessed using a cost-benefit analysis. What were the total costs of the partnership, and what specific costs (transaction costs, operating costs) can be attributed to the partnership? For example, more complex negotiations with a large number of stakeholders initially incur more costs upon the participants, but can later on—in case of successfully institutionalized relationships—lead to considerably lower operating costs. Weakly elaborated contracts between the cooperating parties can result in serious additional costs if the partnership becomes problematic. The extent to which the overall goal of the partnership is aligned with the individual goals of the partners for joining the partnership could also be a fruitful line of enquiry for future research. What critical success factors for managing a partnership do the partners distinguish themselves and how well have they been able to cope with them and learn from it? The efficiency assessment, therefore, contains two specific dimensions: an operational level of project efficiency that links input with output (G1 in Fig. 1) and a tactical level of project performance that links input with outcome (G2).

The effectiveness of partnerships can be seen as the added value and the impact of the partnership compared to individual activities of the different partners. In other words, does the partnership provide additional ways of achieving the societal ambitions that would not have been otherwise possible? Were other objectives possible through the partnership? Were more resources allocated than otherwise possible? Did the partnership project trigger other activities of the participants that proved relevant for obtaining (some of) the societal goals? Is an alternative partnering (or non-partnering) approach possible that would have brought about comparable results? To what extent is the present experience reproducible? What would have happened in case the partnership project was not implemented? The effectiveness question can therefore also be split into two dimensions: a strategic mission-related performance assessment (H1 in Fig. 1), and an issue-related performance measure (H2). The mission-related performance evaluates how the specific partnership made a difference in context and time and as articulated in the partnership’s mission, whereas issue-related performance assesses the contribution of the partnership in providing solution(s) to the initially defined social issue, which might include direct and indirect impacts of a partnership on the issue and in effect re-articulation of the social problem.

Finally, the nature of the issue as well as the degree of efficiency and effectiveness are influenced by the context in which the partnership is initiated. Contexts include various levels of analysis such as: country, region, or global. What might be an effective partnership at the national level might be ineffective at the local or the global levels.

Impact Loops

We can now define four impact loops that can guide further research on cross-sector partnerships impact assessment. Table 1 provides a summary of their most important characteristics. Figure 2 gives a graphical representation.

Table 1 Four orders/loops of partnership impact
Fig. 2
figure 2

Four orders/loops of cross-sector partnership impact

First-order impact loops primarily aim at establishing the impact of partnerships through the effects of internal value-added between inputs (while accounting for costs) and throughputs. A benchmark of success is the operational efficiency attributable to changed inputs and activities, such as greater employee engagement and changed mindsets, for instance. These types of impacts might have further effects on the partners and ultimately the social issue (Austin and Seitanidi 2014; Kolk 2014; Vock et al. 2014; Seitanidi 2009, 2010). The counterfactual is hereby relatively easy to establish by taking other employees that are not involved (or other stakeholders) as a control group or benchmark.

Second-order impact loops capture the effects of internal value added between the inputs and outputs, hence capturing in addition to the operational level effects (first-order impact loop) the tactical level of project performance effects and the interaction between them. Attribution of this effect can in particular be assessed at the output level. Tactical efficiency creates greater project performance by enhanced legitimacy of the project both inside and outside the organization, through institutionalization, realistic contracts, and the creation and implementation of a number of successful partnership management tools (that stimulate learning). The counterfactual is provided by comparing successful and less successful partnerships initiated by the same organization.

Third-order impact loops aim at attributing changed outcomes by capturing the added value of partnerships in the particular context and time of the partnership and according to its mission from inputs to outcomes including the interaction effects across the stages. These effects include synergistic and shared value creation for the participants in the partnership based on mission-related performance. Control groups can be found by comparable partnerships (for instance within the same government subsidy program), by the same partnership over time or by organizations with the same mission definition.

Fourth-order impact effects refer to the overall added value captured by the partnership. It includes all the stages from input to impact and assessing the full extent of the partnership’s contribution to the (social) issue. Fourth-order effects are the most complex to address, because of a large number of levels of analysis, but also due to sizable interaction effects. This effect can be dubbed issue-related performance and the change attributed to partnership involves systemic and societal change. One benchmark of success is the level of innovation that is achieved by the partnership. The counterfactual has to be searched under conditions of a comparable “context”: either in the same country or supply chain in which directly and indirectly involved stakeholders are differently affected by the partnership. An obvious alternative approach is to take a longitudinal perspective and compare “before-after” issue partnerships. For instance, the extent to which the existence of a partnership actually prevented a societal issue from proliferating might be explored.

Impact Orders in the Special Issue

This special issue brings together three largely empirical and one conceptual paper that address the above challenges at varying orders of impact. Each of these contributions introduces new elements to the discussion on impact, at various levels of analysis and with largely complementary theories and methodologies (Table 2). Each of the papers also provides different answers to the various attribution challenges that we specified.

Table 2 Four contributions for this special issue

The paper by Kolk, Vock, and Van Dolen focuses in particular on first-order impact loops. It considers the internal employees of the organizations as the “co-creators” of the partnership and improved CSR strategies. A high level of fit between the core business and the cause increases the willingness of the employees to advocate for the partnership among clients. This fit is influenced by the sector (context) and the type of partnership. Attribution runs through changed mindsets of participants and leaders. The authors label these “trickle effects”. Taking the direction of these trickle effects into consideration presents a technique of assessing in particular positive spill-over and indirect (learning) effects of the partnership. There is no real counterfactual in the paper, although the random sampling selection of the cases amongst pro-active companies in different industries provides a first step (but also a certain sampling bias). The creation of control groups within the same organization provides a logical extension of this line of research.

The paper by Dentoni, Bitzer, and Pascucci concentrates on second-order impact loops (with some reference to first-order loops). It builds on a critical tradition of partnership research and examines the way problem-driven partnerships over time deal with issues in the agro-food industry. It adopts a grounded theory case study approach. By co-creating resources and capabilities in addressing complex problems, parties themselves become beneficiaries. This study looks in particular at how the co-creation of dynamic capabilities changes over time and how this experience has an impact on the partnership: by defining the problem differently (what the authors call a sense-making device and relates to the issue-mission relationship in our model); on stakeholder engagement (which they call higher order dynamic capabilities, i.e., inputs) and on shifting the sustainability goals of companies from reactive to pro-active strategies. The authors are interested in innovative solutions to wicked problems, but argue that as time passes, the partnership effect tends to become lower. The sampling used for this study provides grounds for a counterfactual based on different levels of experience.

The paper by Márquez, Reficco, and Gutiérrez adopts a meso-level of analysis, looking at the portfolio of partnerships, and thus focuses on 2nd (and partly 3rd order) impact loops. The two longitudinal case studies they present take multiple sources of evidence to assess the effects of the partnership. They compare same/intra-sector and cross-sector partnerships, consider the evolution and the extent to which partnership portfolios of companies can be considered homogenous or heterogeneous. Partnerships for the Bottom of the Pyramid are clearly opportunity driven. The degree of success is defined by the reaching of scale of the partnership and ultimately the degree to which the cross-sector partnership is overtaken by same sector partnerships or a go-it-alone strategy. Whether the partnership really contributes to solving the issue (of poverty) is not researched, but can be suggested through enhanced business models in which serving the Bottom of the Pyramid has become normal business practice. The function of the partnership, therefore, is temporary and intended to handle uncertainty and risks associated with entering and creating new markets. The authors do not check whether the non-market partners of the partnership also realize that they engage(d) in a temporary partnership and the degree to which this might have had impact on the activities-output-outcome loop. Linking the partnership portfolio of companies to that of NGOs seems a logical extension of this type of research.

The paper by Stadtler, finally, takes part of the argument that we have developed in this paper one step further by stressing the importance of an actor and stakeholder perspective in assessing impact. The paper focuses in particular on third-order impact loops, by primarily taking the impact value chain as starting position for a broad assessment of the ultimate effects of public–private partnerships. The author provides one illustrative case study, but is primarily interested in defining various levels of analysis and benefits/costs of partnerships. The paper is the only one of the four papers that tries to include benefits as well as costs for society, direct as well as indirect effects for both involved and non-involved stakeholders. So-called “ripple effects” enable greater access to communities and create scale effects. The paper focuses in particular on the relationship between output-outcome-impact at different levels of analysis and stakeholder engagement. The paper looks at the way different partner constellations might affect the ultimate impact for the organizations themselves and for the target group. The nature of the activities is taken as a relative black box, although the effects on the internal stakeholders of the participating organizations are included in the basic framework and related checklist.

All papers define the link with the core strategy of the organizations as particularly relevant for enhanced impact, although most studies also do not empirically cover the ultimate impact of the partnership. Three of the papers include case studies as a comparison, and as partial answer to the challenge of establishing a counterfactual through control groups. But this part of the research is clearly open for improvement. Most papers also take a learning perspective, either through employee engagement (Kolk et al.), issue sense-making (Dentoni et al.) or education (Stadtler). Learning can lead to enhanced output of the partnership, to the longer term survival but also to the termination of the partnership. The papers show that longer-run effects—i.e., taking output and outcome factors into account—can change over time depending on whether the partnership takes an opportunity-driven or a problem-driven road. Ultimately, the four papers of this special issue illustrate the richness of the area, its rapidly growing sophistication, but also illustrate the challenges that are still ahead in further merging the areas of partnership research and impact assessment. A considerable research agenda is carved out for us based on all the contributions of the special issue.