Mapping indicators onto a logic model is obvious – but why haven’t we always done it? August 18, 2009
Posted by Paul Duignan in : Doing evaluation more efficiently, DoView, Easy Outcomes, Indicators, Measurement, Outcomes models, Outcomes theory, Reporting systems, Using the approach , add a commentI was running a workshop today teaching policy analysts the basics of my approach to program evaluation (Easy Outcomes). One of the participants, when I talked about the importance of always mapping indicators back onto a visual model, commented that when you do it, it’s so obviously the right approach that you can’t understand why we’ve not been doing it for years.
The idea behind this approach is that the way we almost always approach indicator work is to eye-ball a list or table of indicators and ask the question of a group of busy people sitting around a table – ‘does this list of indicators look any good?’
(more…)
Tracking jobs created under the U.S. Recovery Act – when should the attempt at measurement be abandoned? June 16, 2009
Posted by Paul Duignan in : Accountability, Attribution, Doing evaluation more efficiently, Evaluation planning, Impact evaluation, Indicators, Measurement, Outcomes systems architecture, Outcomes theory & politics, Outcomes theory & the news, Reporting systems, Using the approach , add a commentThe default expectation in at least some sections of the U.S. public sector seems to be that it should always be feasible and affordable to both measure and attribute the results of interventions. This is using the term attribution to mean being able to actually demonstrate that a change in an outcome has been caused by a particular intervention rather than being the result of other factors (see here for more on attribution). The recent U.S. Recovery Act is a case in point. While it’s reasonable to start from the position that you should routinely assess the possibility of measuring and attributing changes in outcomes of particular interventions, you can’t start by just assuming that it will always be feasible or affordable to do this. Clinging to such an assumption, where it is untrue, can result in you either measuring an outcome when the data you are collecting is not accurate, or acting as though what you are measuring (even if it is an accurate measurement of a change in an outcome) is demonstrably attributable to a particular program, when in fact it may not be. (more…)
The error of limiting focus to only the attributable June 8, 2009
Posted by Paul Duignan in : Accountability, Attribution, Easy Outcomes, Indicators, Measurement, Outcomes systems architecture, Reporting systems, Using the approach , add a commentI am continuing to develop a set of articles which outline various problems which are often built into the outcomes systems which I see. The one I have just put up is on the Error of Limiting Focus to Only the Attributable. This is where the whole emphasis of a performance management or other outcomes system is just on holding a provider to account for a list of demonstrably attributable indicators (often called outputs, deliverables, or key performance indicators). This often creates problems. (more…)
Unalterable deliverables and program inflexibility June 2, 2009
Posted by Paul Duignan in : Accountability, Indicators, Measurement, Outcomes systems architecture, Reporting systems, Using the approach , add a commentBack blogging now after having been on holiday. Recently I ran into the problem of unalterable deliverables in a project I am involved in. This problem was also mentioned in the UN report on its results-based management system that I blogged about a couple of postings ago. The problem arises where a project is set up and deliverables are set, but where ideally there needs to be some flexibility regarding deliverables as the program develops over time. Sometimes the problem is just a result of the difficulty of changing deliverables. (more…)
Intense analysis of the U.N. Results-Based Management System May 5, 2009
Posted by Paul Duignan in : Accountability, Attribution, Doing evaluation more efficiently, Indicators, Measurement, Outcomes models, Outcomes systems architecture, Outcomes theory, Reporting systems, Standards, Using the approach , add a commentI have just put up an Outcomes Theory Knowledge Base article which is an intense analysis of the United Nation Results-Based Management System. (Its obscure work, but someone has to do it!). The exciting part is that it has let me road-test my new Outcomes Systems Checklist. This now provides a common framework for analyzing any outcomes system – outcomes systems being any system which attempts to identify, measure, attribute or hold parties to account for outcomes or the steps which it is thought lead to them. A 2008 report from the U.N. itself on its Results-Based Management System said that the system was: ‘an administrative chore of little value to accountability and decision-making”.
The single list of indicators problem April 27, 2009
Posted by Paul Duignan in : Accountability, Attribution, Indicators, Measurement, Outcomes systems architecture, Reporting systems, Using the approach , 2commentsMany results management, performance management and monitoring systems suffer from what is called the ‘single list of indicators’ problem. I have just put up an article on the Outcomes Theory Knowledge Base regarding this problem (the URL of the article is at the bottom of this blog posting). It arises in situations where there is a demand that an indicator list be high-level (i.e. not at the output level) but at the same time that the list be able to be used to hold a program, organization or other intervention to account. Often one list cannot be used to do both of these jobs. There are four things that can happen in regard to single list approaches, all four create problems and can lead to undermining the credibility of the outcomes system in which they occur. (more…)
Tears in outcomes land: the non-output demonstrably attributable indicator paradox April 23, 2009
Posted by Paul Duignan in : Accountability, Attribution, Indicators, Measurement, Outcomes models, Outcomes systems architecture, Reporting systems, Using the approach , add a commentIf there is one thing that causes a terrible amount of pain in outcomes land, it is the Non-Output Demonstrably Attributable Indicator Paradox. This paradox manifests itself as the demand to find an intermediate outcome which can be used for accountability purposes. The paradox comes into play when this quest is accompanied by a demand that such intermediate outcomes not also be outputs. Sometimes this demand is made explicitly, other times it is made implicitly. Outcomes models which are structured into horizontal layers (outputs, intermediate outcomes, final outcomes), implicitly make this demand by requiring that a step be put in either the outputs or the intermediate outcomes layer within the model. Many funder contract management and provider staff spend hours and hours in rooms trying to find such intermediate outcomes only to walk away frustrated. There is a simple solution to this problem by building a technically sound outcomes system. (more…)
Reliability versus validity – read on it’s important! April 16, 2009
Posted by Paul Duignan in : Accountability, Evaluation debates, Indicators, Measurement , 2commentsNow that Easter is over (and the yard gate has been built to keep in the dog that my wife and the kids have their hearts set on getting). I’m back blogging. Today I want to talk about the difference between reliability and validity. It sounds technical, but read on, its really important in a lot of results and outcomes areas. In psychology, where I come from, they spend a lot of time drumming this distinction into you. Reliability is whether measurements at different times and by different people will give you the same result. Validity is whether you are measuring the right thing. (more…)
Why just about every indicator system in the world needs to be fixed! April 5, 2009
Posted by Paul Duignan in : Accountability, Communicating outcomes models, Doing evaluation more efficiently, Easy Outcomes, Evaluation planning, Indicators, Measurement, Outcomes models, Outcomes systems architecture, Reporting systems, Standards, Using the approach , 1 comment so farI’ve just posted a new article in the Outcomes Theory Knowledge Base on why it is essential to map indicators onto an underlying visual outcomes model. I blogged a little while ago about why we should be wary of too-tidy indicator sets and in the article I explain why. The vast majority of indicator systems in the world suffer from the problem set out in the article – they are just a straight list of indicators set out in tabular format. They give the user no idea as to whether a number of important steps and outcomes are not being measured. Those using such systems remain blissfully unaware of this. In my view, all these straight indicator sets need to be fixed. It’s not particularly difficult, it just requires some work. How to draw the underlying outcomes models is set out in the outcomes model standards and how to then use such models for indicator mapping and many other things is described in detail in the applied version of outcomes theory – Easy Outcomes. (more…)
Beware of suspiciously tidy indicator sets March 19, 2009
Posted by Paul Duignan in : Attribution, Communicating outcomes models, Easy Outcomes, Indicators, Measurement, Outcomes models, Reporting systems , 1 comment so farI’ve just come away from presenting at a national Philanthropy conference as part of a half day session on evaluation and outcomes. I was presenting on the use of the Easy Outcomes approach as a way of grantees structuring outcomes, indicators and evaluation. I will tidy up the outcomes model I used and post a link to it in a blog in a week or so. A lot of interesting points came up in the discussion and I will blog on several of them over the next few days. The first one is to be beware of suspiciously tidy indicator sets. The Easy Outcomes approach gets people to draw an outcomes model (intervention logic) of what they are trying to do without worrying about what they can and can’t measure and what they can, and can’t demonstrate is attributable to their particular project (both of these issues are dealt with later in the process). You draw the models using the guidelines here.
(more…)