Visual model of what I’m trying to do with my outcomes work April 28, 2009
Posted by Paul Duignan in : Blog info, Communicating outcomes models, DoView, Easy Outcomes, Outcomes models, Outcomes theory, Using the approach , 2commentsThought that I would apply a taste of my own medicine to my own work, so I drew a visual outcomes model of what it is that I’m trying to do with my work in the outcomes area. It is here. At the top is my high level outcome ‘Make working with outcomes, monitoring, evaluation etc. easier’ and below that is all of the lower-level steps I am using to get to this high-level outcome. I have included hyperlinks out to the various web sites where I am attempting to do the things listed in the lower-level steps.
Paul Duignan, PhD
Outcomes and Evaluation Blog (OutcomesBlog.org)
The single list of indicators problem April 27, 2009
Posted by Paul Duignan in : Accountability, Attribution, Indicators, Measurement, Outcomes systems architecture, Reporting systems, Using the approach , 2commentsMany results management, performance management and monitoring systems suffer from what is called the ‘single list of indicators’ problem. I have just put up an article on the Outcomes Theory Knowledge Base regarding this problem (the URL of the article is at the bottom of this blog posting). It arises in situations where there is a demand that an indicator list be high-level (i.e. not at the output level) but at the same time that the list be able to be used to hold a program, organization or other intervention to account. Often one list cannot be used to do both of these jobs. There are four things that can happen in regard to single list approaches, all four create problems and can lead to undermining the credibility of the outcomes system in which they occur. (more…)
Don’t assume that impact evaluation should always be done April 26, 2009
Posted by Paul Duignan in : Attribution, Doing evaluation more efficiently, Easy Outcomes, Evaluation planning, Impact evaluation, Outcomes systems architecture , add a commentImpact evaluation – evaluation which looks at whether changes in high-level outcomes can be attributed to a particular program, organization or other intervention – is a particularly useful type of evaluation when done properly. It clearly tells us what works, and what doesn’t work, and this information can be used in decision-making about which programs should, and should not, be funded in the future. However, particularly at the present time, with all of the enthusiasm for evidence-based practice, many people mistakenly assume that impact evaluation should always be attempted in regard to any program, organization or other intervention. Assuming this is a serious mistake. I’ve just put up an article in the Outcomes Theory Knowledge Base which sets out in detail why it is and the way we should approach assessing when impact evaluation should be done. (more…)
Tears in outcomes land: the non-output demonstrably attributable indicator paradox April 23, 2009
Posted by Paul Duignan in : Accountability, Attribution, Indicators, Measurement, Outcomes models, Outcomes systems architecture, Reporting systems, Using the approach , add a commentIf there is one thing that causes a terrible amount of pain in outcomes land, it is the Non-Output Demonstrably Attributable Indicator Paradox. This paradox manifests itself as the demand to find an intermediate outcome which can be used for accountability purposes. The paradox comes into play when this quest is accompanied by a demand that such intermediate outcomes not also be outputs. Sometimes this demand is made explicitly, other times it is made implicitly. Outcomes models which are structured into horizontal layers (outputs, intermediate outcomes, final outcomes), implicitly make this demand by requiring that a step be put in either the outputs or the intermediate outcomes layer within the model. Many funder contract management and provider staff spend hours and hours in rooms trying to find such intermediate outcomes only to walk away frustrated. There is a simple solution to this problem by building a technically sound outcomes system. (more…)
Making outcomes theory more concrete – checklist for assessing outcomes systems April 21, 2009
Posted by Paul Duignan in : Accountability, Attribution, Communicating outcomes models, Doing evaluation more efficiently, Measurement, Outcomes models, Outcomes systems architecture, Outcomes theory, Reporting systems, Strategic planning, Using the approach , 1 comment so farMost normal people would think that it’s very very obscure, but I’ve just put up a Checklist for Analyzing Outcomes Systems in the Outcomes Theory Knowledge Base and it’s a very exciting development. Up until now the Outcomes Theory Knowledge Base has consisted of a set of articles which outline various aspects of outcomes theory. Outcomes theory is a general theory which covers all types of outcomes systems. Outcomes systems are any type of performance management system, results-base system, monitoring system, evaluation system, outcomes-focused contracting system, or strategic planning system (the term even includes evidence-based practice systems). Such systems have, in the past, been seen as somewhat different types of things without a common theory existing to analyze them. Outcomes theory is based on the insight that we can theorize them as a common type of system and then use the theory to work out how such systems should be best structured. This approach becomes powerful at the moment that we can start applying it to actual real-world outcomes systems. This is the role of the checklist I’ve just developed. (more…)
What we are all on about – representing causal models April 19, 2009
Posted by Paul Duignan in : Communicating outcomes models, Doing evaluation more efficiently, Outcomes models, Outcomes systems architecture, Strategic planning, Use of terms, Using the approach , add a commentWhether we know it or not, a lot of us in evaluation, monitoring, social programs, philanthropy etc. spend a lot of time working with ‘causal models’. We call them all sorts of things – program justifications, rationales, program activities and objectives, logic models, logframes, intervention logics, strategy maps etc. – and most people who work with them don’t think of them as causal models. But that’ s what they are if we see causal models as just being an attempt to set out ‘what it is believed causes what in the world’. In the case of a program, the model is going to be a model of the steps which you think a program needs to take in order to cause high-level outcomes to occur. We really should get our heads around the best way to represent such models because at the moment I think that there is a great deal of wasted talk and effort about all of this. And it distracts us from getting on with the job of implementing good programs as fast as possible. Every dollar or every hour spent on struggling with an inefficient way of representing our program is a dollar or hour wasted. (more…)
Reliability versus validity – read on it’s important! April 16, 2009
Posted by Paul Duignan in : Accountability, Evaluation debates, Indicators, Measurement , 2commentsNow that Easter is over (and the yard gate has been built to keep in the dog that my wife and the kids have their hearts set on getting). I’m back blogging. Today I want to talk about the difference between reliability and validity. It sounds technical, but read on, its really important in a lot of results and outcomes areas. In psychology, where I come from, they spend a lot of time drumming this distinction into you. Reliability is whether measurements at different times and by different people will give you the same result. Validity is whether you are measuring the right thing. (more…)
Problems in pay for performance systems April 9, 2009
Posted by Paul Duignan in : Accountability, Attribution, Measurement, Outcomes models, Outcomes systems architecture, Outcomes theory, Reporting systems, Using the approach , 2commentsSorry, I stopped blogging there for a day or two due to a computer problem, and I will also not be blogging over the Easter Break, but will be back daily blogging after that. Today I’m looking at problems in a pay for performance system. From the point of view of outcomes theory, pay for performance systems are just another example of an outcomes system. Outcomes systems are any system which attempts to identify, measure, attribute and hold people, organizations or programs to account. A U.S. GAO report [2] on one such system is interesting reading (for those with a taste for obscure government reports). It reviews the National Security Personnel System which has just been put on hold by the Obama administration and they may axe it [1]. The problems identified in the GAO report include: (more…)
Social Innovation, evaluation and outcomes April 6, 2009
Posted by Paul Duignan in : DoView, Easy Outcomes, Impact evaluation, Outcomes models, Outcomes theory, Outcomes theory & politics, Research influening policy, Using the approach , add a commentI attended a launch of the New Zealand national Center for Social Innovation last night. Geoff Mulgan from the Young Foundation (a similar center in the U.K.) talked about social innovation. The social innovation movement is about getting stakeholders and sectors together to do things differently to achieve better social outcomes. Already a dynamic movement, it has recently received a shot in the arm from the global economic melt-down – traditional ways of doing things are increasingly being questioned and people are looking for new solutions. A number of points made by Geoff and in the subsequent discussion are particularly relevant to outcomes and evaluation were: (more…)
Why just about every indicator system in the world needs to be fixed! April 5, 2009
Posted by Paul Duignan in : Accountability, Communicating outcomes models, Doing evaluation more efficiently, Easy Outcomes, Evaluation planning, Indicators, Measurement, Outcomes models, Outcomes systems architecture, Reporting systems, Standards, Using the approach , 1 comment so farI’ve just posted a new article in the Outcomes Theory Knowledge Base on why it is essential to map indicators onto an underlying visual outcomes model. I blogged a little while ago about why we should be wary of too-tidy indicator sets and in the article I explain why. The vast majority of indicator systems in the world suffer from the problem set out in the article – they are just a straight list of indicators set out in tabular format. They give the user no idea as to whether a number of important steps and outcomes are not being measured. Those using such systems remain blissfully unaware of this. In my view, all these straight indicator sets need to be fixed. It’s not particularly difficult, it just requires some work. How to draw the underlying outcomes models is set out in the outcomes model standards and how to then use such models for indicator mapping and many other things is described in detail in the applied version of outcomes theory – Easy Outcomes. (more…)