Beware of suspiciously tidy indicator sets March 19, 2009
Posted by Paul Duignan in : Attribution, Communicating outcomes models, Easy Outcomes, Indicators, Measurement, Outcomes models, Reporting systems , trackbackI’ve just come away from presenting at a national Philanthropy conference as part of a half day session on evaluation and outcomes. I was presenting on the use of the Easy Outcomes approach as a way of grantees structuring outcomes, indicators and evaluation. I will tidy up the outcomes model I used and post a link to it in a blog in a week or so. A lot of interesting points came up in the discussion and I will blog on several of them over the next few days. The first one is to be beware of suspiciously tidy indicator sets. The Easy Outcomes approach gets people to draw an outcomes model (intervention logic) of what they are trying to do without worrying about what they can and can’t measure and what they can, and can’t demonstrate is attributable to their particular project (both of these issues are dealt with later in the process). You draw the models using the guidelines here.
Once you have drawn the model you then place indicators next to the steps and outcomes in the model that can currently be measured. This immediately makes very clear both the steps and outcomes for which you do not have indicators and also the level at which any indicator sits within the outcomes model. Another approach, which does not place indicators against a visual outcomes model is to just develop a stand-alone set of indicators. The tidiest version of this was one I saw once which was, from memory, for child welfare outcomes in the United Kingdom. It consisted of five headings, with five indicators under each heading. Beware of such indicator lists – they are likely to be too tidy to actually represent the real situation in regard to what you can, and can’t, measure.
As I’ve said, when you use the Easy Outcomes approach you usually find that there are some gaps in your indicator set and that the indicators you have are at various levels within the outcomes model. The problem with a ‘too tidy’ indicator list is that the reality of what you can, and cannot measure, is completely hidden from you. You have no way of knowing whether indicators in the tidy indicator list you are looking are at a high level within the outcomes model or not. So the tidy indicator list could, if you are lucky represent a set of indicators at a high level across your undrawn model, or they could, equally, represent a mess of indicators some at high levels and some at very low levels (often called surrogate indicators or proxy indicators). I think that it is far better to use an approach like Easy Outcomes which forces people to be transparent about what is being measured and the level within an outcomes model at which an indicator sits. The too tidy indicator list hides all this and forces you down the road of just doing the easily measurable rather than attempting to do the important and having measurement as a secondary tool which serves that purpose.
[Later addition: There is now an Outcomes Theory Knowledge Base article setting out this argument. It is Duignan, P. (2009). Indicators – why they should be mapped onto a visual outcomes model. Outcomes Theory Knowledge Base Article No 235. (http://knol.google.com/k/~/~/2m7zd68aaz774/72).
Paul Duignan, PhD
Comments»
[...] to map indicators onto an underlying visual outcomes model. I blogged a little while ago about why we should be wary of too-tidy indicator sets and in the article I explain why. The vast majority of indicator systems in the world suffer from [...]