Getting outcomes creds and saving time! September 25, 2009
Posted by Paul Duignan in : Accountability, Doing evaluation more efficiently, DoView, Easy Outcomes, Outcomes models, Outcomes systems architecture, Philanthropy, Standards, Strategic planning, Using the approach , add a commentPublic sector organizations these days have two important imperatives: establishing that they are truly ‘results and outcomes-focused’ while also becoming more efficient in their internal organizational activity. The really good news in the outcomes area is that using a central tool of outcomes work – outcomes models (a particular type of visual model of all of the high-level outcomes the organization is seeking to achieve and the steps it is taking to do so) is that organizations and programs can do both at the same time. (more…)
The Taxi Driver and ‘why don’t you just measure outcomes’ – on the way to AES conference September 1, 2009
Posted by Paul Duignan in : Doing evaluation more efficiently, Easy Outcomes, Measurement, Outcomes models, Outcomes systems architecture, Reporting systems, Using the approach , add a commentOn my way to the Australasian Evaluation Society Conference in Canberra my taxi driver in from the airport asked me what I do. When I explained that I ‘measure whether programs, often government programs, work or not so the taxpayer gets value for money’, he was right into the concept. Although I think he thought that I was over complicating things a little. He said: ‘shouldn’t it just be a matter of using statistics to measure if things are getting better or not about a program.’ What he was talking about was one aspect of monitoring and evaluation – an important piece – but just one of the Five Building Blocks I see lying behind all monitoring and evaluation systems (outcomes systems). (more…)
Can an exhaustive list of impact evaluation designs be developed, or is my mission on this futile? August 27, 2009
Posted by Paul Duignan in : Attribution, Doing evaluation more efficiently, Easy Outcomes, Evaluation debates, Evaluation planning, Impact evaluation, Outcomes systems architecture, Outcomes theory, Using the approach , add a commentI have set out on a mission as a part of outcomes theory to attempt to develop an exhaustive list of impact/outcome evaluation designs – evaluation designs which make a claim that changes in high-level outcomes can be attributed to a particular intervention. If we could pull off developing such a list that most people are happy with, it would be very powerful. First it could be used in evaluation planning to work out if all of the possible impact evaluation designs had been assessed for their appropriateness, feasibility and/or affordability. At the moment I think that almost every evaluation planner walks around wondering if there is some sort of impact evaluation design they have not considered.
(more…)
Randomistas Rule August 16, 2009
Posted by Paul Duignan in : Attribution, Doing evaluation more efficiently, Evaluation debates, Impact evaluation, Outcomes systems architecture, Outcomes theory, Using the approach , add a commentJust read and commented on an interesting article referred to on the 3IE site – a site dedicated to improving evidence about what works in international development. The article was by Martin Ravallion and was about the rise of the Randomistas in international development economics. Randomistas are those who promote much more use of randomized trials to try and work out what works in international development. It is a good article which points out the fact that randomized trials are not feasible in many important types of development interventions. This debate is the same one which is occurring in many sectors at the moment and one which has been debated on and off in the evaluation field for many years. My take on it is that we need to develop some underlying principle which we can debate and generally agree on so that we don’t need to have this debate endlessly without seemingly making much progress on it.
(more…)
Damascus – YEN Evaluation Clinic August 12, 2009
Posted by Paul Duignan in : Doing evaluation more efficiently, Evaluation planning, Impact evaluation, Outcomes systems architecture, Using the approach , 2commentsApologies for not blogging for a while, I’ve been involved in considerable travel and lots of other work – but that’s really no excuse. Maybe I just got all blogged out. What with Knolling, Blogging here and Twittering, maybe it all just got too much. Anyway, I’m back in the saddle now as they say! Last month I was fortunate to be an evaluation expert at the YEN Evaluation Clinic in Damascus. YEN is the Youth Employment Network – an International Labour Organization, World Bank, United Nations collaboration. A site for the evaluation clinic has been set up at yenclinic.groupsite.co.
The Evaluation Clinic took two examples of youth employment programs and worked through designing an impact evaluation for them. It was a fascinating experience. I’ll blog about what it was like being the sole psychologist evaluator working with five economists evaluation specialists (from the ILO and the World Bank) another day! (more…)
The good old ‘one pager’ contraint again July 9, 2009
Posted by Paul Duignan in : Communicating outcomes models, Doing evaluation more efficiently, DoView, Easy Outcomes, Outcomes models, Outcomes systems architecture, Using the approach , add a commentI’ve been up against the good old ‘one pager’ constraint again in a couple of contexts recently. This is where there is the demand that the outcomes for an organization or program be ‘put on one page’. It comes in various forms and is often a demand from senior management or a perceived demand from them to ‘keep things simple’. Now, there’s nothing wrong with the idea of summarizing things and paper overviews play a role in that. But such things should be seen as one of a range of different types of summaries and products which are produced by an underlying outcomes system, not the beginning and end of an outcomes effort. (more…)
Tracking jobs created under the U.S. Recovery Act – when should the attempt at measurement be abandoned? June 16, 2009
Posted by Paul Duignan in : Accountability, Attribution, Doing evaluation more efficiently, Evaluation planning, Impact evaluation, Indicators, Measurement, Outcomes systems architecture, Outcomes theory & politics, Outcomes theory & the news, Reporting systems, Using the approach , add a commentThe default expectation in at least some sections of the U.S. public sector seems to be that it should always be feasible and affordable to both measure and attribute the results of interventions. This is using the term attribution to mean being able to actually demonstrate that a change in an outcome has been caused by a particular intervention rather than being the result of other factors (see here for more on attribution). The recent U.S. Recovery Act is a case in point. While it’s reasonable to start from the position that you should routinely assess the possibility of measuring and attributing changes in outcomes of particular interventions, you can’t start by just assuming that it will always be feasible or affordable to do this. Clinging to such an assumption, where it is untrue, can result in you either measuring an outcome when the data you are collecting is not accurate, or acting as though what you are measuring (even if it is an accurate measurement of a change in an outcome) is demonstrably attributable to a particular program, when in fact it may not be. (more…)
Impact evaluation on full program roll-out versus just on piloting – two paradigms June 10, 2009
Posted by Paul Duignan in : Attribution, Doing evaluation more efficiently, Easy Outcomes, Impact evaluation, Outcomes systems architecture, Using the approach , add a commentI’ve just posted an article on the two paradigms in regard to impact/outcome evaluation and full program roll-out. What this is about is making a distinction between designing an evaluation which can provide impact/outcome evaluation information about full program roll-out versus a paradigm where you do impact/outcome evaluation just on piloting and then in regard to full program roll-out you just make sure that best practice is implemented. I once was involved in the evaluation of an overall program which had over 900 component programs. The way that we went about evaluating it was, in my view, wrong. (more…)
The error of limiting focus to only the attributable June 8, 2009
Posted by Paul Duignan in : Accountability, Attribution, Easy Outcomes, Indicators, Measurement, Outcomes systems architecture, Reporting systems, Using the approach , add a commentI am continuing to develop a set of articles which outline various problems which are often built into the outcomes systems which I see. The one I have just put up is on the Error of Limiting Focus to Only the Attributable. This is where the whole emphasis of a performance management or other outcomes system is just on holding a provider to account for a list of demonstrably attributable indicators (often called outputs, deliverables, or key performance indicators). This often creates problems. (more…)
Over-simplifications in outcomes, monitoring and evaluation June 3, 2009
Posted by Paul Duignan in : Accountability, Attribution, Doing evaluation more efficiently, Easy Outcomes, Evaluation planning, Outcomes systems architecture, Outcomes theory, Reporting systems, Using the approach , add a commentAn evaluation colleague Patrica Rogers commented on an earlier blog posting of mine in which I was claiming that what I am trying to do it to make outcomes, monitoring and evaluation work ‘easier’. She challenged me on that idea and pointed out that often what we are having to deal with is over-simplification in the way people are working with outcomes, monitoring and evaluation. Her comment inspired me to work up an article on over-simplification in outcomes and evaluation and after getting underway with it I realized all of the different ways in which people approach outcomes, monitoring and evaluation with over-simplified approaches and the problems which these cause. (more…)