Program theory evaluation: Logic analysis

Abstract

Program theory evaluation, which has grown in use over the past 10 years, assesses whether a program is designed in such a way that it can achieve its intended outcomes. This article describes a particular type of program theory evaluation—logic analysis—that allows us to test the plausibility of a program's theory using scientific knowledge. Logic analysis is useful for improving the intervention or finding alternatives for achieving intended outcomes; it influences the choice of type of evaluation to conduct and strengthens the validity of subsequent evaluations.

The objective of this article is to present the methodological principles and the roots of this type of program theory evaluation. We illustrate two types of logic analysis with two actual evaluation cases. There are very few published examples of program theory evaluation. This article will provide evaluators with both theoretical and practical information to help them in conceptualizing their evaluations.

Introduction

“Programs are complicated phenomena, generally born out of experience and professional lore” (Weiss, 1998). When requesting an evaluation, stakeholders generally want to know if what they are doing works and how they might improve their intervention. Program theory evaluation can often provide that kind of information without mobilizing the research efforts of an effect analysis.

Weiss defines program theory as “the mechanisms that mediate between the delivery (and receipt) of the program and the emergence of the outcomes of interest” (Weiss, 1998). Logic modelling is generally presented in the evaluation literature as a way to open the black box to better understand finer causal mechanisms. Some sophisticated logic modelling approaches include evidence-based depictions of the causes of the targeted problem, to better appreciate the intervention's potential impact (Renger and Hurley, 2006, Renger and Titcomb, 2002). While this is an important step in better understanding the action mechanisms of the intervention, it is still not enough. As evaluators, we should also question the validity of the intervention's chain of action (validity of the means), and we should test the scientific plausibility of the program's theory. In fact, it could be argued that program theory does not really reflect how the intervention produces the intended outcomes, but rather, stakeholders’ perceptions and beliefs, right or wrong, about the mechanisms that operate between the delivery of the intervention and the intended outcomes. The whole evaluation is then built on the consensus reached on stakeholders’ beliefs and perceptions. But what do we do as evaluators if these are incomplete or, worse, if they are wrong? Can we really build valid evaluations based on the prior analysis of a program's theory that reflects what people think, but not what the intervention does (Chen, 1990a, Chen, 1990b)?

As we will argue here, testing the program's theory before entering more deeply into the evaluation process can provide important insights into the validity of the intervention's means of action while mobilizing stakeholders in a valuable exercise of reflection (Pawson et al., 2005, Renger and Hurley, 2006, Page et al., 2007). It can also help in choosing the type of subsequent evaluation that will best suit the context and the intervention's characteristics. There is an important body of documented experience of program theory evaluation (Campbell, 1966, Chen and Rossi, 1987, Lipsey, 1989, Marquart, 1990, Smith, 1990, Weitzman et al., 2002). Several approaches and methodologies have been explored (Bickman, 1987a, Conrad and Miller, 1987, Cook, 2000, Lipsey, 1993, Weiss, 2000). Yet, in most cases important dilemmas were raised and significant evaluation effort was expended (Weiss, 1997), which may help explain why program theory evaluation is not more systematically used. In this article, we describe our experience of a certain type of program theory evaluation called logic analysis (Champagne et al., 2009a, Contandriopoulos et al., 2000). Logic analysis is a program theory evaluation based on existing knowledge that uses expert judgment and scientific literature reviews. We begin with a presentation of the conceptual, methodological, and historical aspects of logic analysis. Then we illustrate the discussion with two examples, logic analysis of the Research Collective on Primary Care and logic analysis of integrated services for persons with dual diagnoses of mental health and substance use disorders.