Tracking jobs created under the U.S. Recovery Act - when should the attempt at measurement be abandoned? June 16, 2009
Posted by Paul Duignan in : Impact evaluation, Outcomes systems architecture, Attribution, Reporting systems, Outcomes theory & politics, Indicators, Accountability, Using the approach, Doing evaluation more efficiently, Measurement, Outcomes theory & the news, Evaluation planning , add a commentThe default expectation in at least some sections of the U.S. public sector seems to be that it should always be feasible and affordable to both measure and attribute the results of interventions. This is using the term attribution to mean being able to actually demonstrate that a change in an outcome has been caused by a particular intervention rather than being the result of other factors (see here for more on attribution). The recent U.S. Recovery Act is a case in point. While it’s reasonable to start from the position that you should routinely assess the possibility of measuring and attributing changes in outcomes of particular interventions, you can’t start by just assuming that it will always be feasible or affordable to do this. Clinging to such an assumption, where it is untrue, can result in you either measuring an outcome when the data you are collecting is not accurate, or acting as though what you are measuring (even if it is an accurate measurement of a change in an outcome) is demonstrably attributable to a particular program, when in fact it may not be. (more…)
Extraordinary circumstances and Dick Cheney’s ’stuff happens’ March 29, 2009
Posted by Paul Duignan in : Attribution, Outcomes theory & politics, Evaluation debates, Outcomes theory, Accountability, Outcomes theory & the news , add a commentIn my last blog posting I commented on Jon Stewart’s critique of ex-Vice President Dick Cheney’s claim that the Bush administration should not be held accountable for the U.S. economic melt-down because a number of things happened during their term which affected the economy. The Vice President summarized this by saying that ’stuff happens’ and this ’stuff’ unexpectedly blew their budget. The ’stuff’ included the wars in Afghanistan and Iraq, and Hurricane Katrina. In technical outcomes theory terms, the Vice-President was mounting an ‘extraordinary factors’ argument to reduce his administration’s accountability for the economic melt-down. (more…)
To attribute or not to attribute - Jon Stewart vs Dick Cheney March 27, 2009
Posted by Paul Duignan in : Attribution, Outcomes theory & politics, Evaluation debates, Outcomes theory, Accountability, Outcomes theory & the news , 3commentsIn a recent episode of Jon Stewart’s Daily Show he deconstructs a high-profile interview with Dick Cheney, the previous Vice President of the United States, undertaken by another interviewer. While this is a comedy show, being an overly analytical sort of person, I can’t watch it without analyzing what is going on it in from the point of view of outcomes theory! At a technical level, the key issue Stewart is focusing on in creating his laughs is what in outcomes theory is called - demonstrable attribution. Demonstrable attribution is being able to demonstrate that an improvement which occurs following an intervention has been caused by a particular intervention (see here for more outcomes theory information on this). In summary, Stewart claims that Dick Cheney is applying a double standard around demonstrable attribution. (more…)
Time for an Attribution Commission? March 17, 2009
Posted by Paul Duignan in : Outcomes theory & politics, Attribution, Accountability, Economic analysis, Outcomes theory & the news , 1 comment so farA blog posting by Dean Baker (he is one of the few economists who accurately predicted the housing collapse) looks at the ‘comparative responsibility’ of Bush and Obama for the 2009 U.S. deficit in the face of some Republican claims that somehow it is Obama’s fault. He argues that most of it is the legacy of Bush. Whenever an administration changes you get endless debate as to who is responsible for what. In fact much of political discourse is about attribution and accountability, where attribution is about attempting to demonstrate that an agent caused something to happen and accountability is whether they should be punished or rewarded for what has happened. (See the article here for more on attribution and accountability within outcomes theory). (more…)
Bonuses role in the financial melt-down March 16, 2009
Posted by Paul Duignan in : Outcomes theory, Reporting systems, Outcomes theory & politics, Indicators, Accountability, Measurement, Outcomes theory & the news, Outcomes models , add a commentPresident Obama has amplified the attack on bonuses being paid to staff in companies which have been bailed out by the U.S. government (CNN, 16 March 2009). What does outcomes theory have to say about the role of the bonus system in the current financial meltdown? I blogged in 2007 about the problem of bonuses in the financial system and how it was possible the full extent of the melt-down would take time to be revealled. Thinking in terms of an outcomes model, what has happened is that financial institutions should have been aiming (as they do in healthy times) at the overall outcome of Sustainable long-run profitability. (more…)
14 years for revealing an indicator! February 8, 2009
Posted by Paul Duignan in : Outcomes theory & politics, Indicators, Outcomes theory & the news , 1 comment so farOne of the principles of outcomes theory is that if you want groups of stakeholders to not have a complete understanding of what is happening as an outcomes model plays out, you try to suppress information about indicators of outcomes which are heading in the wrong direction. There is currently a story in the media about a UK officer being arrested in Afghanistan for allegedly supplying civilian casualty figures to a human rights campaigner. The CNN story is here.
More on politicians - accountability (praise and blame) and having ‘blown it’ December 10, 2008
Posted by Paul Duignan in : Evaluation debates, Reporting systems, Outcomes systems architecture, Outcomes theory, Accountability, Indicators, Outcomes theory & the news , add a comment
Further to my last posting on whether or not the jump in President Bush’s approval rating was attributable to him (i.e. provable that he caused most of it, rather than it just being the fact that he happened to find himself being President on September 11th), I came across a letter to Newsweek which relates to another aspect of the features of steps and outcomes in outcomes models as related to politicians. (The letter is the second letter listed here). This time it is the issue of accountability. (more…)
Attibution and President Bush’s approval rating December 7, 2008
Posted by Paul Duignan in : Impact evaluation, Attribution, Reporting systems, Outcomes systems architecture, Indicators, Outcomes theory & the news, Accountability, Measurement , add a comment
I’ve been doing a lot of thinking about indicators and attribution these days. I think that a lot of problems (particularly misinterpretations of how well or badly parties are doing within outcomes systems of all types arise because of confusion between attributable and not-necessarily attributable indicators. The underying Five Building Blocks Diagram which is used in outcomes theory makes a distiction between attributable and not-necessarily attributable indicators.The Washington Post today had an interactive showing President Bush’s Approval Rating over time. At various points, the graph had labels showing what was happening at the time, presumably to help the reader work out why his rating went up and down at the points it did. The biggest jump was around September 11 where his approval took a massive leap from something like 56% to 92% (reading off the graph). Below is a section from the interactive. (more…)
Trading-off indicator accuracy for incentivization - forecast calls for pain October 3, 2007
Posted by Paul Duignan in : Indicators, Outcomes theory & the news , 2comments
An article in the London Times today (4 October 2007) - Why a bulging bonus is part of the problem, argues that the current credit crisis following the melt-down of the US mortgage markets has not yet run its full course. In particular, the problem is that in the global banking system, ‘the losers do not know that they are losers, even now’. It will only be when banks start selling off their investment assets that the real price of these assets will be determined and at that time there’ll be lots more pain. In the meantime, it’s likely that a falsely positive picture will be painted because of the bonus systems for employees in the banking industry. This system means that ‘there is tremendous incentive for everyone from the chief executive down to believe the best when millions in personal rewards are at stake’. As a result they’ll overvalue banks assets in the short term. In the run up to the crisis, the first half year results for banks were the best they’ve ever achieved. In order for employees to get an annual bonus they’ll need to also show good results in the second half of the year or else they’ll lose the potential bonus they have on the books from first half of the year.
This is a classic example of the outcomes theory principle of the trade-off between indicator accuracy and using indicators for incentivization where indicators are easy to falsify. And ‘it is the very nature of the bonus system that may end up prolonging the agony’. Perhaps we could go even further than the article and wonder about the roll of the bonus system not only in prolonging the agony, but as a contributing factor in the creation of the problem in the first place. (more…)
Increased airplane safety by IGNORING final outcomes October 1, 2007
Posted by Paul Duignan in : Accountability, Outcomes theory & the news, Outcomes models , 1 comment so far
The New York Times carries a story today, ‘Fatal Airplane Crashes Drop 65%‘. It reports that fatal airplane crashes in the US have dropped by 65%. The death rate in 1997 was one in 2 million whereas the death rate now is about one in 4.5 million. I think the level of airplane safety is one of the great administrative, regulatory and engineering achievements of our time. It shows what can be done when people are serious about managing negative externalities - bad things that happen in the course of selling goods or services. This success story illustrates an important, but seemingly counterintuitive, principle of outcomes theory - deciding whether or not a decision was a correct one often does not depend on the final outcomes from that decision. In other words, you can improve airline safety by ignoring final outcomes! (more…)