Tears in outcomes land: the non-output demonstrably attributable indicator paradox April 23, 2009
Posted by Paul Duignan in : Accountability, Attribution, Indicators, Measurement, Outcomes models, Outcomes systems architecture, Reporting systems, Using the approach , trackbackIf there is one thing that causes a terrible amount of pain in outcomes land, it is the Non-Output Demonstrably Attributable Indicator Paradox. This paradox manifests itself as the demand to find an intermediate outcome which can be used for accountability purposes. The paradox comes into play when this quest is accompanied by a demand that such intermediate outcomes not also be outputs. Sometimes this demand is made explicitly, other times it is made implicitly. Outcomes models which are structured into horizontal layers (outputs, intermediate outcomes, final outcomes), implicitly make this demand by requiring that a step be put in either the outputs or the intermediate outcomes layer within the model. Many funder contract management and provider staff spend hours and hours in rooms trying to find such intermediate outcomes only to walk away frustrated. There is a simple solution to this problem by building a technically sound outcomes system.
Having an outcomes system which includes this paradox always ends in tears and it can do this in three ways:
- Exhaustion and giving oneself up to incoherency. This is where those looking for the non-output demonstrably attributable indicator, having been unable to find such a thing, simply give up, list anything they like as indicators, and walk away from the problem. This leads to an incoherent outcomes system.
- Relaxing the constraint that the indicator should be demonstrably attributable. This leads to a list of intermediate outcomes which are at a higher level than outputs. However, because they are not demonstrably attributable to the program, organization or intervention which is being held to account for them, this can lead to one of three problems. The first is that where such an indicator is not met, a provider can claim that other factors influenced its lack of achievement and therefore attempt to escape from being held to account for it. The second is that a provider can be unfairly punished when such an indicator is not met, due to its failure to being met having resulted from other factors. The third is that a provider can be unfairly rewarded when such an indicator is met, but it is not due to anything in particular that they have done since it has been influenced by other factors. Over time, the credibility of outcomes systems which suffer from these problems is progressively undermined.
- Relaxing the constrain that the indicator should be a non-output. This leads to the credibility of the system as an outcomes-focused system being undermined as it soon becomes apparent that the accountability indicators which are being used are just set at the outputs level.
This pain and these tears can all be avoided by building a system which conforms to the principles of outcomes theory. I’ve just put up an article on the Outcomes Theory Knowledge Base which describes the paradox and refers to the articles which show how it can be completely avoided just by building a technically sound outcomes system.
Later addition: I now have an article on the non-output attributable intermediate outcome paradox. It is Duignan, P. (2009). Non-output demonstrably attributable indicator paradox. Outcomes Theory Knowledge Base Article No. 241. (http://knol.google.com/k/paul-duignan-phd/non-output-attributable-intermediate/2m7zd68aaz774/83).
Paul Duignan, PhD
Outcomes and Evaluation Blog (OutcomesBlog.org)
Comments»
no comments yet - be the first?