Tears in outcomes land: the non-output demonstrably attributable indicator paradox April 23, 2009
Posted by Paul Duignan in : Accountability, Attribution, Indicators, Measurement, Outcomes models, Outcomes systems architecture, Reporting systems, Using the approach , add a commentIf there is one thing that causes a terrible amount of pain in outcomes land, it is the Non-Output Demonstrably Attributable Indicator Paradox. This paradox manifests itself as the demand to find an intermediate outcome which can be used for accountability purposes. The paradox comes into play when this quest is accompanied by a demand that such intermediate outcomes not also be outputs. Sometimes this demand is made explicitly, other times it is made implicitly. Outcomes models which are structured into horizontal layers (outputs, intermediate outcomes, final outcomes), implicitly make this demand by requiring that a step be put in either the outputs or the intermediate outcomes layer within the model. Many funder contract management and provider staff spend hours and hours in rooms trying to find such intermediate outcomes only to walk away frustrated. There is a simple solution to this problem by building a technically sound outcomes system. (more…)
Making outcomes theory more concrete – checklist for assessing outcomes systems April 21, 2009
Posted by Paul Duignan in : Accountability, Attribution, Communicating outcomes models, Doing evaluation more efficiently, Measurement, Outcomes models, Outcomes systems architecture, Outcomes theory, Reporting systems, Strategic planning, Using the approach , 1 comment so farMost normal people would think that it’s very very obscure, but I’ve just put up a Checklist for Analyzing Outcomes Systems in the Outcomes Theory Knowledge Base and it’s a very exciting development. Up until now the Outcomes Theory Knowledge Base has consisted of a set of articles which outline various aspects of outcomes theory. Outcomes theory is a general theory which covers all types of outcomes systems. Outcomes systems are any type of performance management system, results-base system, monitoring system, evaluation system, outcomes-focused contracting system, or strategic planning system (the term even includes evidence-based practice systems). Such systems have, in the past, been seen as somewhat different types of things without a common theory existing to analyze them. Outcomes theory is based on the insight that we can theorize them as a common type of system and then use the theory to work out how such systems should be best structured. This approach becomes powerful at the moment that we can start applying it to actual real-world outcomes systems. This is the role of the checklist I’ve just developed. (more…)
Problems in pay for performance systems April 9, 2009
Posted by Paul Duignan in : Accountability, Attribution, Measurement, Outcomes models, Outcomes systems architecture, Outcomes theory, Reporting systems, Using the approach , 2commentsSorry, I stopped blogging there for a day or two due to a computer problem, and I will also not be blogging over the Easter Break, but will be back daily blogging after that. Today I’m looking at problems in a pay for performance system. From the point of view of outcomes theory, pay for performance systems are just another example of an outcomes system. Outcomes systems are any system which attempts to identify, measure, attribute and hold people, organizations or programs to account. A U.S. GAO report [2] on one such system is interesting reading (for those with a taste for obscure government reports). It reviews the National Security Personnel System which has just been put on hold by the Obama administration and they may axe it [1]. The problems identified in the GAO report include: (more…)
Simplifying outcomes terminology – angels dancing on the head of a pin! April 1, 2009
Posted by Paul Duignan in : Accountability, Attribution, Communicating outcomes models, Doing evaluation more efficiently, DoView, Easy Outcomes, Outcomes theory, Reporting systems, Using the approach , add a commentIn the middle ages, churchmen occupied themselves by having long discussions sitting around the fire drinking fine wine and discussing how many angels could dance on the head of a pin. It was tough, but someone had to do it! That was until someone called William of Occam came along. He is famous for spoiling the party with something called ‘Occam’s Razor’ – one way of describing the way he operated was that the did not want people to ‘multiply entities beyond necessity’ (see Wikipedia entry). What he was talking about was trying to keep things as simple as possible. This is the approach which is taken within outcomes theory. (more…)
Extraordinary circumstances and Dick Cheney’s ‘stuff happens’ March 29, 2009
Posted by Paul Duignan in : Accountability, Attribution, Evaluation debates, Outcomes theory, Outcomes theory & politics, Outcomes theory & the news , add a commentIn my last blog posting I commented on Jon Stewart’s critique of ex-Vice President Dick Cheney’s claim that the Bush administration should not be held accountable for the U.S. economic melt-down because a number of things happened during their term which affected the economy. The Vice President summarized this by saying that ‘stuff happens’ and this ‘stuff’ unexpectedly blew their budget. The ‘stuff’ included the wars in Afghanistan and Iraq, and Hurricane Katrina. In technical outcomes theory terms, the Vice-President was mounting an ‘extraordinary factors’ argument to reduce his administration’s accountability for the economic melt-down. (more…)
To attribute or not to attribute – Jon Stewart vs Dick Cheney March 27, 2009
Posted by Paul Duignan in : Accountability, Attribution, Evaluation debates, Outcomes theory, Outcomes theory & politics, Outcomes theory & the news , 3commentsIn a recent episode of Jon Stewart’s Daily Show he deconstructs a high-profile interview with Dick Cheney, the previous Vice President of the United States, undertaken by another interviewer. While this is a comedy show, being an overly analytical sort of person, I can’t watch it without analyzing what is going on it in from the point of view of outcomes theory! At a technical level, the key issue Stewart is focusing on in creating his laughs is what in outcomes theory is called – demonstrable attribution. Demonstrable attribution is being able to demonstrate that an improvement which occurs following an intervention has been caused by a particular intervention (see here for more outcomes theory information on this). In summary, Stewart claims that Dick Cheney is applying a double standard around demonstrable attribution. (more…)
Beware of suspiciously tidy indicator sets March 19, 2009
Posted by Paul Duignan in : Attribution, Communicating outcomes models, Easy Outcomes, Indicators, Measurement, Outcomes models, Reporting systems , 1 comment so farI’ve just come away from presenting at a national Philanthropy conference as part of a half day session on evaluation and outcomes. I was presenting on the use of the Easy Outcomes approach as a way of grantees structuring outcomes, indicators and evaluation. I will tidy up the outcomes model I used and post a link to it in a blog in a week or so. A lot of interesting points came up in the discussion and I will blog on several of them over the next few days. The first one is to be beware of suspiciously tidy indicator sets. The Easy Outcomes approach gets people to draw an outcomes model (intervention logic) of what they are trying to do without worrying about what they can and can’t measure and what they can, and can’t demonstrate is attributable to their particular project (both of these issues are dealt with later in the process). You draw the models using the guidelines here.
(more…)
Time for an Attribution Commission? March 17, 2009
Posted by Paul Duignan in : Accountability, Attribution, Economic analysis, Outcomes theory & politics, Outcomes theory & the news , 1 comment so farA blog posting by Dean Baker (he is one of the few economists who accurately predicted the housing collapse) looks at the ‘comparative responsibility’ of Bush and Obama for the 2009 U.S. deficit in the face of some Republican claims that somehow it is Obama’s fault. He argues that most of it is the legacy of Bush. Whenever an administration changes you get endless debate as to who is responsible for what. In fact much of political discourse is about attribution and accountability, where attribution is about attempting to demonstrate that an agent caused something to happen and accountability is whether they should be punished or rewarded for what has happened. (See the article here for more on attribution and accountability within outcomes theory). (more…)
What can be claimed about whether a program works or not from a logic model? January 13, 2009
Posted by Paul Duignan in : Attribution, Communicating outcomes models, Impact evaluation, Outcomes models, Uncategorized , 1 comment so far
We sometimes hear things like: “a logic model was used to show that the program works”. I’m interested in tidying up such talk so that we are very clear about exactly what it is that is being claimed in regard to showing whether or not a program works. I’ve put up a new article in Knol which deals with the question of the types of claims we can make in regard to logic models (I call them outcomes models) and the types of arguments we can mount about whether or not programs work based on these claims. The article is here.The article is rather intense, but I think that it’s important that, as evaluators, we get on top of this sort of thing. In the article, I set out three claims that can be made in regard to logic models (or sub-parts of logic models). These are: (more…)
Ease of impact evaluation and bias within ‘what works’ evidence-based practice systems January 4, 2009
Posted by Paul Duignan in : Attribution, Evaluation planning, Impact evaluation, Outcomes systems architecture, Outcomes theory, Using the approach , add a comment
Back on deck after Christmas and turning the mind back from eating and having a great time with relatives to outcomes and evaluation again. I’ve just put up an article in the Outcomes Theory Knowledge Base about the issue of evidence-based practice systems not taking into account variations in the ease of undertaking impact evaluation between program types.
There are many examples of evidence-based practice systems being set up (two classic examples are the Cochrane Collaboration in health care and the Campbell Collaboration in the social policy area) which are attempting to use information from impact evaluation (evaluation which demonstrates attribution of improvements in high level outcomes to particular programs). I think that such initiatives are generally a great idea. My article deals with a particular situation which sometimes arises which needs to be addressed in any well constructed ‘what works’ system. (more…)