Mapping indicators onto a logic model is obvious – but why haven’t we always done it? August 18, 2009
Posted by Paul Duignan in : Doing evaluation more efficiently, DoView, Easy Outcomes, Indicators, Measurement, Outcomes models, Outcomes theory, Reporting systems, Using the approach , add a commentI was running a workshop today teaching policy analysts the basics of my approach to program evaluation (Easy Outcomes). One of the participants, when I talked about the importance of always mapping indicators back onto a visual model, commented that when you do it, it’s so obviously the right approach that you can’t understand why we’ve not been doing it for years.
The idea behind this approach is that the way we almost always approach indicator work is to eye-ball a list or table of indicators and ask the question of a group of busy people sitting around a table – ‘does this list of indicators look any good?’
(more…)
Randomistas Rule August 16, 2009
Posted by Paul Duignan in : Attribution, Doing evaluation more efficiently, Evaluation debates, Impact evaluation, Outcomes systems architecture, Outcomes theory, Using the approach , add a commentJust read and commented on an interesting article referred to on the 3IE site – a site dedicated to improving evidence about what works in international development. The article was by Martin Ravallion and was about the rise of the Randomistas in international development economics. Randomistas are those who promote much more use of randomized trials to try and work out what works in international development. It is a good article which points out the fact that randomized trials are not feasible in many important types of development interventions. This debate is the same one which is occurring in many sectors at the moment and one which has been debated on and off in the evaluation field for many years. My take on it is that we need to develop some underlying principle which we can debate and generally agree on so that we don’t need to have this debate endlessly without seemingly making much progress on it.
(more…)
Damascus – YEN Evaluation Clinic August 12, 2009
Posted by Paul Duignan in : Doing evaluation more efficiently, Evaluation planning, Impact evaluation, Outcomes systems architecture, Using the approach , 2commentsApologies for not blogging for a while, I’ve been involved in considerable travel and lots of other work – but that’s really no excuse. Maybe I just got all blogged out. What with Knolling, Blogging here and Twittering, maybe it all just got too much. Anyway, I’m back in the saddle now as they say! Last month I was fortunate to be an evaluation expert at the YEN Evaluation Clinic in Damascus. YEN is the Youth Employment Network – an International Labour Organization, World Bank, United Nations collaboration. A site for the evaluation clinic has been set up at yenclinic.groupsite.co.
The Evaluation Clinic took two examples of youth employment programs and worked through designing an impact evaluation for them. It was a fascinating experience. I’ll blog about what it was like being the sole psychologist evaluator working with five economists evaluation specialists (from the ILO and the World Bank) another day! (more…)
The good old ‘one pager’ contraint again July 9, 2009
Posted by Paul Duignan in : Communicating outcomes models, Doing evaluation more efficiently, DoView, Easy Outcomes, Outcomes models, Outcomes systems architecture, Using the approach , add a commentI’ve been up against the good old ‘one pager’ constraint again in a couple of contexts recently. This is where there is the demand that the outcomes for an organization or program be ‘put on one page’. It comes in various forms and is often a demand from senior management or a perceived demand from them to ‘keep things simple’. Now, there’s nothing wrong with the idea of summarizing things and paper overviews play a role in that. But such things should be seen as one of a range of different types of summaries and products which are produced by an underlying outcomes system, not the beginning and end of an outcomes effort. (more…)
Tracking jobs created under the U.S. Recovery Act – when should the attempt at measurement be abandoned? June 16, 2009
Posted by Paul Duignan in : Accountability, Attribution, Doing evaluation more efficiently, Evaluation planning, Impact evaluation, Indicators, Measurement, Outcomes systems architecture, Outcomes theory & politics, Outcomes theory & the news, Reporting systems, Using the approach , add a commentThe default expectation in at least some sections of the U.S. public sector seems to be that it should always be feasible and affordable to both measure and attribute the results of interventions. This is using the term attribution to mean being able to actually demonstrate that a change in an outcome has been caused by a particular intervention rather than being the result of other factors (see here for more on attribution). The recent U.S. Recovery Act is a case in point. While it’s reasonable to start from the position that you should routinely assess the possibility of measuring and attributing changes in outcomes of particular interventions, you can’t start by just assuming that it will always be feasible or affordable to do this. Clinging to such an assumption, where it is untrue, can result in you either measuring an outcome when the data you are collecting is not accurate, or acting as though what you are measuring (even if it is an accurate measurement of a change in an outcome) is demonstrably attributable to a particular program, when in fact it may not be. (more…)
Impact evaluation on full program roll-out versus just on piloting – two paradigms June 10, 2009
Posted by Paul Duignan in : Attribution, Doing evaluation more efficiently, Easy Outcomes, Impact evaluation, Outcomes systems architecture, Using the approach , add a commentI’ve just posted an article on the two paradigms in regard to impact/outcome evaluation and full program roll-out. What this is about is making a distinction between designing an evaluation which can provide impact/outcome evaluation information about full program roll-out versus a paradigm where you do impact/outcome evaluation just on piloting and then in regard to full program roll-out you just make sure that best practice is implemented. I once was involved in the evaluation of an overall program which had over 900 component programs. The way that we went about evaluating it was, in my view, wrong. (more…)
Over-simplifications in outcomes, monitoring and evaluation June 3, 2009
Posted by Paul Duignan in : Accountability, Attribution, Doing evaluation more efficiently, Easy Outcomes, Evaluation planning, Outcomes systems architecture, Outcomes theory, Reporting systems, Using the approach , add a commentAn evaluation colleague Patrica Rogers commented on an earlier blog posting of mine in which I was claiming that what I am trying to do it to make outcomes, monitoring and evaluation work ‘easier’. She challenged me on that idea and pointed out that often what we are having to deal with is over-simplification in the way people are working with outcomes, monitoring and evaluation. Her comment inspired me to work up an article on over-simplification in outcomes and evaluation and after getting underway with it I realized all of the different ways in which people approach outcomes, monitoring and evaluation with over-simplified approaches and the problems which these cause. (more…)
Being sector-centric not program-centric in deciding on program evaluation priorities May 22, 2009
Posted by Paul Duignan in : Doing evaluation more efficiently, Evaluation planning, Impact evaluation , add a commentI have been blogging less in the last week or so due to going on holiday – blogging will still be less frequent for a week or so. I was recently involved in assessing a set of projects summaries to help determine which of them should be selected for more intensive evaluation input. This was not being done directly to determine project evaluation priorities, however the exercise reminded me once again of the general issue of how we determine what types of evaluation should be undertaken for particular projects. My blog comments below are about the general issue rather than the particular exercise of project selection I was recently involved in. The set of information we typically use to work out what type of evaluation should be undertaken for a project is information such as the nature of the project, the proposed evaluation questions, and the proposed evaluation budget. I think that we need more than this when determining evaluation priorities. (more…)
Intense analysis of the U.N. Results-Based Management System May 5, 2009
Posted by Paul Duignan in : Accountability, Attribution, Doing evaluation more efficiently, Indicators, Measurement, Outcomes models, Outcomes systems architecture, Outcomes theory, Reporting systems, Standards, Using the approach , add a commentI have just put up an Outcomes Theory Knowledge Base article which is an intense analysis of the United Nation Results-Based Management System. (Its obscure work, but someone has to do it!). The exciting part is that it has let me road-test my new Outcomes Systems Checklist. This now provides a common framework for analyzing any outcomes system – outcomes systems being any system which attempts to identify, measure, attribute or hold parties to account for outcomes or the steps which it is thought lead to them. A 2008 report from the U.N. itself on its Results-Based Management System said that the system was: ‘an administrative chore of little value to accountability and decision-making”.
Don’t assume that impact evaluation should always be done April 26, 2009
Posted by Paul Duignan in : Attribution, Doing evaluation more efficiently, Easy Outcomes, Evaluation planning, Impact evaluation, Outcomes systems architecture , add a commentImpact evaluation – evaluation which looks at whether changes in high-level outcomes can be attributed to a particular program, organization or other intervention – is a particularly useful type of evaluation when done properly. It clearly tells us what works, and what doesn’t work, and this information can be used in decision-making about which programs should, and should not, be funded in the future. However, particularly at the present time, with all of the enthusiasm for evidence-based practice, many people mistakenly assume that impact evaluation should always be attempted in regard to any program, organization or other intervention. Assuming this is a serious mistake. I’ve just put up an article in the Outcomes Theory Knowledge Base which sets out in detail why it is and the way we should approach assessing when impact evaluation should be done. (more…)