Moving past the debate about randomized experiments May 25, 2010
Posted by Paul Duignan in : Impact evaluation, Evaluation debates, Doing evaluation more efficiently, Using the approach, Evaluation planning, Easy Outcomes , 2commentsA colleague Bob Williams recently drew attention to articles on the New Yorker about the use of randomized experiments and particularly one from an economist advocating their widespread use in a range of program areas.
I’ve been involved in a number of seemingly endless discussions and presentations about the pros and cons of randomized experiments and the rise of what are being called the Randomistas - those advocating for a much wider use of randomized experiments. In this post I want to talk about how we can move beyond these seemingly endless discussions. (more…)
Theory of Change Versus Theory of Action January 15, 2010
Posted by Paul Duignan in : Attribution, Outcomes theory, Doing evaluation more efficiently, Using the approach, Standards , 2commentsWhat’s the difference between a Theory of Change and a Theory of Action? I’m just clarifying my thoughts on this issue and how it relates to my work thinking about how we conceptualize outcomes models (logic models) within outcomes theory. In summary, at the moment - apart from a Theory of Action just being an outcomes model drawn at a lower level - I can’t see a major difference. However I’m happy to be contradicted on this and will change my view if there are convincing arguments for making the distinction. My current thinking is as set out below. (more…)
The evolution of the logic model January 13, 2010
Posted by Paul Duignan in : Communicating outcomes models, Doing evaluation more efficiently, Using the approach, Standards, Outcomes models, DoView , 4commentsI’ve just posted an article on the evolution of the logic model within evaluation. Over the last couple of decades, increasing numbers of evaluators have started using logic models. For those not familiar with what logic models are - they are simply tabular or visual representations of all of the lower-level steps needed to achieve high-level outcomes for a program, organization or other intervention. They go by different names, for instance: program logics, intervention logics, results maps, theories of change, program theories, results hierarchies, strategy maps, end-means diagrams etc.). A traditional way of drawing logic models has evolved (known as the inputs, outputs, intermediate outcomes, final outcomes structured logic model) which often attempts to restrict logic models to a single page. However, many evaluators are now breaking away from the constraints of this traditional format and exploring various alternative ways of representing logic models. (more…)
Developing an M&E plan using a visual approach January 8, 2010
Posted by Paul Duignan in : Doing evaluation more efficiently, Using the approach, Evaluation planning, Easy Outcomes, DoView , add a commentOn various lists I am on I often see requests by people wanting to develop what is called an M&E plan. This terminology is often used in the international development area. It refers to a Monitoring and Evaluation Plan. The way these requests are made makes me think that the way you should monitor and evaluate different projects varies a great deal. (more…)
Getting outcomes creds and saving time! September 25, 2009
Posted by Paul Duignan in : Doing evaluation more efficiently, Accountability, Outcomes systems architecture, Philanthropy, Strategic planning, Using the approach, Easy Outcomes, Outcomes models, Standards, DoView , add a commentPublic sector organizations these days have two important imperatives: establishing that they are truly ‘results and outcomes-focused’ while also becoming more efficient in their internal organizational activity. The really good news in the outcomes area is that using a central tool of outcomes work - outcomes models (a particular type of visual model of all of the high-level outcomes the organization is seeking to achieve and the steps it is taking to do so) is that organizations and programs can do both at the same time. (more…)
Using an outcomes modeling approach to action research September 9, 2009
Posted by Paul Duignan in : Doing evaluation more efficiently, Using the approach, Outcomes models, Easy Outcomes , add a commentWill get back to blogging on the Australasian Evaluation Society Conference when I get a moment (may not be for a few days). In the meantime had to prepare an article about using outcomes modeling as a basic tool within an action research approach. Because outcomes modeling - developing visual outcomes models (like a type of logic model, or theory of change model) - according to the outcomes theory set of standards for building such models is a generic process, such models can be used for a wide range of purposes. They can, for instance, be used within an action research approach. Action research is an approach which attempts to work in cycles of research/action/research. It has the great virtue of ensuring that research is connected to action and action is connected to research.
(more…)
The Taxi Driver and ‘why don’t you just measure outcomes’ - on the way to AES conference September 1, 2009
Posted by Paul Duignan in : Outcomes systems architecture, Reporting systems, Measurement, Doing evaluation more efficiently, Outcomes models, Using the approach, Easy Outcomes , add a commentOn my way to the Australasian Evaluation Society Conference in Canberra my taxi driver in from the airport asked me what I do. When I explained that I ‘measure whether programs, often government programs, work or not so the taxpayer gets value for money’, he was right into the concept. Although I think he thought that I was over complicating things a little. He said: ’shouldn’t it just be a matter of using statistics to measure if things are getting better or not about a program.’ What he was talking about was one aspect of monitoring and evaluation - an important piece - but just one of the Five Building Blocks I see lying behind all monitoring and evaluation systems (outcomes systems). (more…)
Can an exhaustive list of impact evaluation designs be developed, or is my mission on this futile? August 27, 2009
Posted by Paul Duignan in : Impact evaluation, Evaluation debates, Attribution, Outcomes systems architecture, Outcomes theory, Evaluation planning, Using the approach, Doing evaluation more efficiently, Easy Outcomes , add a commentI have set out on a mission as a part of outcomes theory to attempt to develop an exhaustive list of impact/outcome evaluation designs - evaluation designs which make a claim that changes in high-level outcomes can be attributed to a particular intervention. If we could pull off developing such a list that most people are happy with, it would be very powerful. First it could be used in evaluation planning to work out if all of the possible impact evaluation designs had been assessed for their appropriateness, feasibility and/or affordability. At the moment I think that almost every evaluation planner walks around wondering if there is some sort of impact evaluation design they have not considered.
(more…)
Untangling evaluation terms - discussing evaluation ‘types’ with clients often more useful than evaluation ‘approaches’ August 24, 2009
Posted by Paul Duignan in : Use of terms, Evaluation debates, Doing evaluation more efficiently, Using the approach, Evaluation planning , add a commentI have just put up a outcomes theory article based on a book chapter I wrote some time ago dividing the terminology used in evaluation into five groups of terms about five different ‘aspects’ of evaluation. These aspects are: evaluation approaches; evaluation types (based on the purpose of the evaluation); evaluation methods; evaluation information analysis techniques; and evaluation designs. Approaches tend to combine a range of different elements including general approaches to evaluation, philosophy of science views and for instance, quasi-political perspectives on the relationship between empowered and disempowered groups. Evaluation approaches are often not mutually exclusive from each other from a conceptual point of view. Evaluation approaches include such things as Scriven’s Goal Free Evaluation, Patton’s Utilization Focused Evaluation and Fetterman’s Empowerment Evaluation. While I find these very interesting from the point of view of stimulating my thinking about evaluation, I often (but not always) do not find them very useful when talking to a client about a specific evaluation.
(more…)
Formative evaluation versus impact/outcome evaluation August 20, 2009
Posted by Paul Duignan in : Evaluation debates, Use of terms, Outcomes theory, Doing evaluation more efficiently, Evaluation planning, Using the approach, Easy Outcomes , add a commentIn response to a posting on one of my outcomes theory articles by Marcus Pilgrim who ran the recent YEN Evaluation Clinic in Damascus, I have worked up an article on the difference between formative, process and impact/outcome evaluation. As Marcus points out in his posting, the term formative (or developmental) evaluation is not one which is widely known in all sectors. Formative evaluation is directed at optimizing program implementation. Process evaluation attempts to describe the course and context of a program. Impact/outcome evaluation looks at the intended and unintended, positive and negative outcomes of a program and whether they can be attributed to the program. (more…)