Mapping indicators onto a logic model is obvious – but why haven’t we always done it? August 18, 2009
Posted by Paul Duignan in : Doing evaluation more efficiently, DoView, Easy Outcomes, Indicators, Measurement, Outcomes models, Outcomes theory, Reporting systems, Using the approach , trackbackI was running a workshop today teaching policy analysts the basics of my approach to program evaluation (Easy Outcomes). One of the participants, when I talked about the importance of always mapping indicators back onto a visual model, commented that when you do it, it’s so obviously the right approach that you can’t understand why we’ve not been doing it for years.
The idea behind this approach is that the way we almost always approach indicator work is to eye-ball a list or table of indicators and ask the question of a group of busy people sitting around a table – ‘does this list of indicators look any good?’
As a psychologist, I’ve always thought that this is an incredibly difficult cognitive task. Normally you get most people just saying ‘well it looks OK to me.’ What people faced with this task are doing in their heads, is some version of an exercise where they try to create a model of the outcomes for the program or organization and then try to work out how well the list of indicators measures these outcomes.
Of course, it’s very difficult to do this just in your head, and when you have a lot of people all sitting around a table with potentially different models they have conjured up in their heads, it is really a recipe for confusion.
So when you simply map proposed indicators back onto a visual model, as I now recommend we should always do, it all seems so much simpler. You can immediately see what you can, and cannot, currently measure (i.e. those outcomes which do not have an indicator next to them are the ones you currently cannot measure). You can also get a very clear picture of what the situation is in regard to proxy-indicators. Proxy-indicators are used in situations where you don’t have an indicator at the highest-level within an outcomes model so you move back down the model to locate the first place you come to where you do have an indicator and you use that as a proxy measurement for the high-level outcome.
Of course, the problem in doing this when you have not mapped your indicators back onto a visual outcomes model is that you then have no feel for the level at which different indicators sit. So on one side of a model you may be measuring high up the outcomes model, right at high-level outcomes or almost at that level. On another side of the model, however, you may be using a proxy-indicator which is a long way down the model. You get no feel of this from a straight indicator list or an indicator table.
Mapping indicators back onto a visual outcomes model makes it immediately obvious where different indicators are located within the model. Of course, many people don’t want to face the reality of what their indicator sets really look like. They would rather have a tidy indicator set in a list or in a table and don’t really want to have rubbed in their face the fact that their measurement of what is happening in the outcomes model is patchy and many of their indicators are down below high-level outcomes.
My feeling on this is that it is unprofessional to not face up to the reality of what we are dealing with in indicator work. Stakeholders, particularly high-level stakeholders, should not be shielded from the reality of how much uncertainty there is in regard to the indicators they are using to make strategic decisions. To do so is to expose them to risk they do not understand they are being exposed to because of the false sense of security a tidy indicator list gives them.
Apart from what you might term psychological resistance to actually laying bare what the situation is in regard to an indicator set, I think that the major stumbling block that has prevented us visualizing indicators mapped back onto a model in the past is much more prosaic. It is simply that without the ability to draw outcomes models in a way that leaves enough space to put in indicators, the task of mapping indicators back onto a model becomes painfully difficult. Outcomes models (logic models) which are crammed onto a single page and which rely on line and arrow links simple have no space left where you can clearly visualize indicators next to the steps and outcomes they measure.
Prior to being involved in developing DoView visual planning and evaluation software, I used to use a numbering system where I gave steps and outcomes within a model numbers and then used these to refer to the indicators which were measuring those steps or outcomes. You can seen me using this approach here (not in regard to indicators this time, but a similar approach with evaluation questions mapped onto an outcomes model. However such an approach, because the user has to repeatedly look at the diagram and then back at the table, really does not give them a quick feel for how indicators actually map onto the outcomes model.
Anyway, now with DoView, it’s trivial to map indicators back onto the relevant outcomes model. Of course, you need to be clear about the outcomes model drawing standards you’re using to draw your outcomes model. If you have drawn an outcomes model which only includes the measurable, then obviously you are not going to find any situations where there is no indicator next to a step or outcome, because any such (not currently measured) steps and outcomes have, by definition, already been excluded from the model.
Anyway, there’s a full article about why indicators should always be mapped onto a visual model here.
Paul Duignan, PhD (Outcomes and evaluation blog OutcomesBlog.org)
Follow me on Twitter or subscribe to my E-Newsletter.
Comments»
no comments yet - be the first?