The single list of indicators problem April 27, 2009
Posted by Paul Duignan in : Accountability, Attribution, Indicators, Measurement, Outcomes systems architecture, Reporting systems, Using the approach , trackbackMany results management, performance management and monitoring systems suffer from what is called the ‘single list of indicators’ problem. I have just put up an article on the Outcomes Theory Knowledge Base regarding this problem (the URL of the article is at the bottom of this blog posting). It arises in situations where there is a demand that an indicator list be high-level (i.e. not at the output level) but at the same time that the list be able to be used to hold a program, organization or other intervention to account. Often one list cannot be used to do both of these jobs. There are four things that can happen in regard to single list approaches, all four create problems and can lead to undermining the credibility of the outcomes system in which they occur.
The first is that people just give up and the system becomes a confused mess with no one really having any idea of whether or not the elements in the list should or should not be things that program, organization or intervention should be held to account for. The second is that, in order to deal with the problem, the indicators become what are called compound indicators – these are ones which include both a high-level outcome and an output. This creates technical problems which are described in the new article. The third is that people give up on the idea that the indicators should be able to be demonstrated as attributable to a particular intervention. This leads to various problems. The fourth is that people give up on the idea that the list should not contain outputs. This leads to disappointment that the system is no longer outcomes focused.
All of this pain and unhappiness can be avoided by setting the system up in the correct way in the first place. The key point is that you need two lists (or, what amounts to the same thing, two types of indicators). The first type is those that are demonstrably attributable – their mere measurement establishes that they have been caused by a particular program, organization or intervention – these are used for accountability purposes. The second type is those which are non-demonstrably attributable – these can be used for strategic purposes to see how outcomes are tracking. How to set up a system which does not suffer from any of these problems is set out in Outcomes Theory Knowledge Base article on Contracting for outcomes.
The single list article is below, if you have any comments on it, put them up at the bottom of the article.
Duignan, P. (2009).Single list of indicators problem. Outcomes Theory Knowledge Base Article No. 243. (http://knol.google.com/k/paul-duignan-phd/single-list-of-indicators-problem/2m7zd68aaz774/85)
Paul Duignan, PhD
Outcomes and Evaluation Blog (OutcomesBlog.org)
Comments»
Swine Flu is an interesting example of an intervention with quite tangled outcomes and a weak indicator set. I realised this on reading my Mike Cosgraves reflection on it (http://www.mikecosgrave.com/blog2006/?p=223).
There is a tangled mess of indicators for Swine flu – WHO pandemic alert level, mortality, infection rate and so forth. We can think of Swine Flu as a sort of ‘antisocial intervention’, with an intervention logic, outcomes model and indicators of it’s own. It has strategic outcomes (‘make people sick’ or ‘kill people’) but the set of indicators given to us via the media, as observers, provides little clear sense of how likely it is to achieve it’s strategic outcomes, or what kinds of decisions we should me making about that.
Of course, thinking about Swine Flu as an intervention reminds me that Outcome theory and tools can be used both ways – as a tool for mapping a programme logic of negative things, war, plague, poverty and so on, it can guide us to develop meaningful indicators to monitor these things better. By describing these things accurately in terms of cause and effect, it may guide is in how to ‘break’ these negative interventions – a sort of formative evaluation in reverse.
Thanks for this posting Robert, I checked out Mike’s article and will do a blog posting on it.