The Taxi Driver and ‘why don’t you just measure outcomes’ – on the way to AES conference September 1, 2009
Posted by Paul Duignan in : Doing evaluation more efficiently, Easy Outcomes, Measurement, Outcomes models, Outcomes systems architecture, Reporting systems, Using the approach , trackbackOn my way to the Australasian Evaluation Society Conference in Canberra my taxi driver in from the airport asked me what I do. When I explained that I ‘measure whether programs, often government programs, work or not so the taxpayer gets value for money’, he was right into the concept. Although I think he thought that I was over complicating things a little. He said: ‘shouldn’t it just be a matter of using statistics to measure if things are getting better or not about a program.’ What he was talking about was one aspect of monitoring and evaluation – an important piece – but just one of the Five Building Blocks I see lying behind all monitoring and evaluation systems (outcomes systems).
In technical terms what he was talking about are what are called not-necessarily demonstrably attributable indicators (I often shorten the term to something like tracking high-level outcome trends – although this term does not differentiate between those that can be attributed and those that cannot by merely measuring them). These are measures of whether or not things are getting better over time. For instance I’ve just this minute emailed someone involved in the Healthcare Information for All by 2015 (HIFA2015) project on exactly this issue. The project is directed at improving the access of healthcare workers throughout the world to relevant healthcare information. A non-necessarily demonstrably attributable indicator for this project would be whether by 2015 the amoung of relevant information healthcare workers have has increased. This is a non-necessarily demonstrably attributable indicator because its mere measurement does not establish that HIFA2015 caused this to happen. Even if there was no HIFA2015, other things being equal, we would probably expect the amount of relevant information available to healthcare workers to have increased by 2015. If we want to establish if HIFA2015 made an impact we are going to have to go beyond just tracking trends in high-level indicators. This is not to say that tracking such trends is not very infomative in terms of whether the overall strategic objective (getting more healthcare workers with relevant information) is being achieved. Doing impact evaluation is another one of the Five Building Blocks.
Such indicators are contrasted with demonstrably attributable indicators – ones which, by their mere measurement, are taken to demonstrate that they have been caused by a particular program. For instance, in the case of HIFA2015 these would be things like the number of people involved in their electronic e-mail forums or the number of postings on those forums etc. These are often referred to as indicators related to outputs. In addition to these two types of indicators (which can be thought of as the ‘monitoring’ aspect in monitoring and evaluation) there are two types of evaluation included in the Five Building Blocks approach – impact evaluation and non-impact evaluation (formative and process evaluation). These four Building Blocks are all integrated onto the first building block – an outcomes model (often called a logic model).
Skimming through the program for the AES conference preparing for a paper I’m giving tomorrow on: ‘A concise framework for thinking about the types of evidence provided by monitoring and evaluation’ I came across another example in a presentation by Margaret Thomas and Wendy Oakes on ‘Evaluation design for a complex tobacco control program – tension between ‘outcomes’ and ‘evidence’. In their abstract they write:
While quit rates are the most commonly reported measure for the success of tobacco control interventions, this was not a feasible and practical focus for the evaluation of this program. Success at lower levels of the outcomes hierarchy was an important ‘outcome’ for [the program] but would not necessarily provide acceptable ‘evidence’ of success to academics and policy makers. This presentation will describe how this tension is being played out in practice.
They are grappling with exactly the issue that the Five Building Blocks approach is designed to assist with. This tension exists beneath many, if not most, evaluations. One danger, is that by defining ‘success at lower levels of the outcomes hierarchy’ as an important ‘outcome’ of the program, somehow changes in high-level outcomes smoking quit rates in this case will somehow not be seen as an ‘outcome’ of the program. I am not saying that Thomas and Oakes are going down this line because I have not heard their presentation. It is just that it often occurs in other evaluations – evaluators and others try to keep a tight reign on what will be regarded as program ‘outcomes’ and try to exclude those at a higher-level which are going to prove hard to attribute to a program program because of the particular circumstances of that program and its nature.
The best way I have found of dealing with this issue and managing the ‘tension being played out in practice’ as they put it, is to always talk in terms of the Five Building Blocks whenever discussing the program and its evaluation. When you do this there is really no conceptual difficulties and no need to try and control how people use the term outcome in regard to different levels within the outcomes hierarchy. Using the Five Building Blocks approach you can measure what you like at any level, for high-level outcomes these measurements will often be of not-necessarily attributable indicators. You can then measure lower level indicators which are attributable and you can do what you can do in regard to impact evaluation and non-impact evaluation and map this all back onto the outcomes model to lock it all together. A paper on the Five Building Blocks can be found here. A paper on the use of language in evaluation (in the specific case of building outcomes models) can be found here.
Anyway, looking through the program there are lots of interesting presentations at the conference, if I get a moment will blog on more of it.
Paul Duignan, PhD. (Follow me on my Outcomes Blog; Twitter; or via my E-Newsletter).
Comments»
no comments yet - be the first?