Theory of Change Versus Theory of Action January 15, 2010
Posted by Paul Duignan in : Attribution, Doing evaluation more efficiently, Outcomes theory, Standards, Using the approach , 2commentsWhat’s the difference between a Theory of Change and a Theory of Action? I’m just clarifying my thoughts on this issue and how it relates to my work thinking about how we conceptualize outcomes models (logic models) within outcomes theory. In summary, at the moment – apart from a Theory of Action just being an outcomes model drawn at a lower level – I can’t see a major difference. However I’m happy to be contradicted on this and will change my view if there are convincing arguments for making the distinction. My current thinking is as set out below. (more…)
Can an exhaustive list of impact evaluation designs be developed, or is my mission on this futile? August 27, 2009
Posted by Paul Duignan in : Attribution, Doing evaluation more efficiently, Easy Outcomes, Evaluation debates, Evaluation planning, Impact evaluation, Outcomes systems architecture, Outcomes theory, Using the approach , add a commentI have set out on a mission as a part of outcomes theory to attempt to develop an exhaustive list of impact/outcome evaluation designs – evaluation designs which make a claim that changes in high-level outcomes can be attributed to a particular intervention. If we could pull off developing such a list that most people are happy with, it would be very powerful. First it could be used in evaluation planning to work out if all of the possible impact evaluation designs had been assessed for their appropriateness, feasibility and/or affordability. At the moment I think that almost every evaluation planner walks around wondering if there is some sort of impact evaluation design they have not considered.
(more…)
Randomistas Rule August 16, 2009
Posted by Paul Duignan in : Attribution, Doing evaluation more efficiently, Evaluation debates, Impact evaluation, Outcomes systems architecture, Outcomes theory, Using the approach , add a commentJust read and commented on an interesting article referred to on the 3IE site – a site dedicated to improving evidence about what works in international development. The article was by Martin Ravallion and was about the rise of the Randomistas in international development economics. Randomistas are those who promote much more use of randomized trials to try and work out what works in international development. It is a good article which points out the fact that randomized trials are not feasible in many important types of development interventions. This debate is the same one which is occurring in many sectors at the moment and one which has been debated on and off in the evaluation field for many years. My take on it is that we need to develop some underlying principle which we can debate and generally agree on so that we don’t need to have this debate endlessly without seemingly making much progress on it.
(more…)
Tracking jobs created under the U.S. Recovery Act – when should the attempt at measurement be abandoned? June 16, 2009
Posted by Paul Duignan in : Accountability, Attribution, Doing evaluation more efficiently, Evaluation planning, Impact evaluation, Indicators, Measurement, Outcomes systems architecture, Outcomes theory & politics, Outcomes theory & the news, Reporting systems, Using the approach , add a commentThe default expectation in at least some sections of the U.S. public sector seems to be that it should always be feasible and affordable to both measure and attribute the results of interventions. This is using the term attribution to mean being able to actually demonstrate that a change in an outcome has been caused by a particular intervention rather than being the result of other factors (see here for more on attribution). The recent U.S. Recovery Act is a case in point. While it’s reasonable to start from the position that you should routinely assess the possibility of measuring and attributing changes in outcomes of particular interventions, you can’t start by just assuming that it will always be feasible or affordable to do this. Clinging to such an assumption, where it is untrue, can result in you either measuring an outcome when the data you are collecting is not accurate, or acting as though what you are measuring (even if it is an accurate measurement of a change in an outcome) is demonstrably attributable to a particular program, when in fact it may not be. (more…)
Impact evaluation on full program roll-out versus just on piloting – two paradigms June 10, 2009
Posted by Paul Duignan in : Attribution, Doing evaluation more efficiently, Easy Outcomes, Impact evaluation, Outcomes systems architecture, Using the approach , add a commentI’ve just posted an article on the two paradigms in regard to impact/outcome evaluation and full program roll-out. What this is about is making a distinction between designing an evaluation which can provide impact/outcome evaluation information about full program roll-out versus a paradigm where you do impact/outcome evaluation just on piloting and then in regard to full program roll-out you just make sure that best practice is implemented. I once was involved in the evaluation of an overall program which had over 900 component programs. The way that we went about evaluating it was, in my view, wrong. (more…)
The error of limiting focus to only the attributable June 8, 2009
Posted by Paul Duignan in : Accountability, Attribution, Easy Outcomes, Indicators, Measurement, Outcomes systems architecture, Reporting systems, Using the approach , add a commentI am continuing to develop a set of articles which outline various problems which are often built into the outcomes systems which I see. The one I have just put up is on the Error of Limiting Focus to Only the Attributable. This is where the whole emphasis of a performance management or other outcomes system is just on holding a provider to account for a list of demonstrably attributable indicators (often called outputs, deliverables, or key performance indicators). This often creates problems. (more…)
Over-simplifications in outcomes, monitoring and evaluation June 3, 2009
Posted by Paul Duignan in : Accountability, Attribution, Doing evaluation more efficiently, Easy Outcomes, Evaluation planning, Outcomes systems architecture, Outcomes theory, Reporting systems, Using the approach , add a commentAn evaluation colleague Patrica Rogers commented on an earlier blog posting of mine in which I was claiming that what I am trying to do it to make outcomes, monitoring and evaluation work ‘easier’. She challenged me on that idea and pointed out that often what we are having to deal with is over-simplification in the way people are working with outcomes, monitoring and evaluation. Her comment inspired me to work up an article on over-simplification in outcomes and evaluation and after getting underway with it I realized all of the different ways in which people approach outcomes, monitoring and evaluation with over-simplified approaches and the problems which these cause. (more…)
Intense analysis of the U.N. Results-Based Management System May 5, 2009
Posted by Paul Duignan in : Accountability, Attribution, Doing evaluation more efficiently, Indicators, Measurement, Outcomes models, Outcomes systems architecture, Outcomes theory, Reporting systems, Standards, Using the approach , add a commentI have just put up an Outcomes Theory Knowledge Base article which is an intense analysis of the United Nation Results-Based Management System. (Its obscure work, but someone has to do it!). The exciting part is that it has let me road-test my new Outcomes Systems Checklist. This now provides a common framework for analyzing any outcomes system – outcomes systems being any system which attempts to identify, measure, attribute or hold parties to account for outcomes or the steps which it is thought lead to them. A 2008 report from the U.N. itself on its Results-Based Management System said that the system was: ‘an administrative chore of little value to accountability and decision-making”.
The single list of indicators problem April 27, 2009
Posted by Paul Duignan in : Accountability, Attribution, Indicators, Measurement, Outcomes systems architecture, Reporting systems, Using the approach , 2commentsMany results management, performance management and monitoring systems suffer from what is called the ‘single list of indicators’ problem. I have just put up an article on the Outcomes Theory Knowledge Base regarding this problem (the URL of the article is at the bottom of this blog posting). It arises in situations where there is a demand that an indicator list be high-level (i.e. not at the output level) but at the same time that the list be able to be used to hold a program, organization or other intervention to account. Often one list cannot be used to do both of these jobs. There are four things that can happen in regard to single list approaches, all four create problems and can lead to undermining the credibility of the outcomes system in which they occur. (more…)
Don’t assume that impact evaluation should always be done April 26, 2009
Posted by Paul Duignan in : Attribution, Doing evaluation more efficiently, Easy Outcomes, Evaluation planning, Impact evaluation, Outcomes systems architecture , add a commentImpact evaluation – evaluation which looks at whether changes in high-level outcomes can be attributed to a particular program, organization or other intervention – is a particularly useful type of evaluation when done properly. It clearly tells us what works, and what doesn’t work, and this information can be used in decision-making about which programs should, and should not, be funded in the future. However, particularly at the present time, with all of the enthusiasm for evidence-based practice, many people mistakenly assume that impact evaluation should always be attempted in regard to any program, organization or other intervention. Assuming this is a serious mistake. I’ve just put up an article in the Outcomes Theory Knowledge Base which sets out in detail why it is and the way we should approach assessing when impact evaluation should be done. (more…)