Sharing the pain - crazy indicators, targets and funder reporting systems December 4, 2008
Posted by Paul Duignan in : Indicators, Outcomes systems architecture, Reporting systems, Accountability, Measurement, Easy Outcomes, Using the approach, Doing evaluation more efficiently, DoView , trackback
This blog post is a follow-up to an earlier posting on my blog. In a comment on that posting, M&Egirl outlined problems she was having with a funder demanding targets that are meaningless and asked for more comment on this obsession with targets on the part of funders.
The program she is working on is a Gender Related Violence program, which for the sake of this discussion I’ll presume involves women who have been subject to gender related violence receiving some sort of intervention to help them be safe and move beyond the trauma they have suffered. Because the issue of indicators and targets is such an important issue I thought that I’d do a further posting on it. If you have a moment, have a quick read of the original posting and M&Egirl’s comment before reading this blog posting.
Obsessions with targets and other reporting demands that funders make on providers are part of the architecture of the outcomes system they’ve set up (more on what I mean by an outcomes system is here). The funder’s problem is obviously to figure out if the provider is doing the types of things that will improve high-level outcomes. Such reporting is set up in a variety of ways using concepts such as deliverables, indicators, targets etc and requirements that these been set out in lists and tables of various sorts. Just before we start, if you are not familiar with it have a quick look at the Five Building Blocks of All Outcomes Systems which is the framework I use to talk about these things - I find unless we are clear about what we mean by indicators, evaluation etc, in a discussion like this we can get a bit lost.
The problem with the way most outcomes systems are set up is that they’re based on a deludedly oversimplified view of the real world in which such outcomes systems have to operate. This view is that, in general - 1) all important outcomes can be measured (have indicators); 2) there is some way of striking realistic targets on all such indicators; and 3) that it is relatively easy to attribute changes in high-level indicators to the actual actions of a provider. I call this the ‘Perfect Knowledge Fallacy’.
None of these things are usually true in the real-world programs we have to work with. However, the funder does not want to believe this. If it is true don’t know how they’re then going to be able to find out whether or not what the provider is doing is likely to improve high-level outcomes. They have unrealistic pressures on them from higher-up that they should have some way of having perfect knowledge about what their providers are doing. Providers go along with this fiction because they often feel that their inability to provide the information that funders want somehow reflects that they are not smart enough to give the funder what they want in regard to their particular program. This is made worse by other providers appearing to be able to play the game, It is also worsened because the successful examples which are provided as to how one should do it, where they are provided, are all cases where this kind of stuff can be done relatively easily. Road safety is a classic example of where the Perfect Knowledge Fallacy problems are not so bad, so performance management systems from that sector are often held up as examples. For instance, here is a recent publication which includes reference to some work I had some involvement in in the road safety area as such an example.
Where it can be done, it is obviously great to do it. However, where there are significant uncertainties involved in monitoring, performance management specialists and evaluators also sometimes go along with the Perfect Knowledge Falacy because they don’t feel confident enough to tell it like it is. What they should be telling funders is something like - ‘Funders, in this case, there’s lots of stuff you’re not going to know, if you don’t like this - just get over it. We’ll help you as best we can to deal with this uncertainity, but beware if you don’t understand the level of uncertainity you are dealing with, you will set up incoherent systems which cause you and your providers all sorts of technical difficulties and will waste a lot of time and cause all concerned unnecessary angst.’
M&Egirl’s funder’s rigid insistence that she provide a target in a case where she thinks it is meaningless is a good example of one of these technical problems. I can’t emphasis enough how much time is wasted on this sort of technical issue which stems directly from an outcomes system being set up which has serious flaws. How to set up systems which do not have these kinds of flaws is set out in my Knol article on Contracting for Outcomes.
How can we deal with the Perfect Knowledge Falacy? The only way I know of doing this is what I call ‘Sharing the Pain’. We need to share the pain about what is and what is not knowable in the case of a particular program with all of the stakeholders concerned. In particular, providers need to share it with their funders as funders are the ones usually calling the shots when it comes to the way in which reporting has been set up within an outcomes system.
The problem is, of course, how best to share the pain, as quickly and cleanly as possible and in a way that makes it obvious to all concerned. The danger is that the provider gets into long explanations - rather like the kind of thing I’m setting out here. This is OK for backgrounding the issue and increasing awareness amongst funders and others over the course of time. But the problem is that when a particular provider does this in the case of a particular program, e.g. M&Egirl says to her funder ‘I can’t give you a target on that’ and tries to explain why, it often all ends up sounding like special pleading on her part because people all over town are feeding the funder with lists of indicators and targets. Of course, the quality of those targets and the basis for them is something which most people don’t have the time or much in the way of a conceptual framework to assess.
I have come to the conclusion that there’s a basic presentational problem with almost all of the traditional ways of structuring reporting information. Anything that relies on tables and lists is very likely to run into problems as soon as things start getting the least bit complex in regard to knowability.
Therefore my work has focused on developing tools and approaches for the use of visual outcomes models which I’ve now concluded are the most effective way of presenting the true situation in regard to knowability - and hence sharing the pain as quickly and easily as possible. The great advantage with sharing the pain in this way is that most of it is self-evident once you start working with the visual outcomes model itself - so the provider does not have to spell it all out, the funder just comes to realize it as they are using the visual model.
How you can do this in contracting is spelt out in my Contracting for Outcomes article.
I have mocked-up a DoView example how M&Egirl could use a visual model to talk to her funder. The web page version of the model is here. This model is just an illustrative mock-up and may not represent how she sees the program, however she could change it in any way to make it look more like her program (M&Egirl you can just grab the model from the website (click on Download the DoView File of This Model in the options bar at the bottom of the model), download a trial copy of DoView and start playing with it).
However M&Egirl may want to structure her model, if she uses this visual approach she now has an environment in which she can deal with her funder and quickly share with them the pain she’s facing over the difficulty of measuring aspects of her program. The things she can do with this model when interacting with her funder, if it is drawn the type of way I have drawn it in the mock-up (see the tips and standards for doing this) are as follows:
1. Make the point that not every important outcome may be able to be measured (this is done by keeping outcomes separate from indicators in your visual outcomes model - it is then immediately obvious which ones are, and which are not, able to be measured at the current time). An example of a currently unmeasured outcome in the model is ‘Women not stigmatized by the process’ - it is presumably an important outcome, but no one has yet figured out a way of measuring it (this may or may not be the case in reality). (To see the page with this outcome on it look here). This makes the point visually that there are some outcomes which all may agree are important, but which are currently not measured). There is no way of doing this if you are just working off a list of measurable indicators or just allowed to set numerical targets.
2. Show the provider all the relevant outcomes and indicators. If M&Egirl just tells her funder ‘not I’m not going to give you a target on that’ the funder immediately feels lost, what are they going to use to keep track of whether M&Egirl’s program is tracking as it should? If M&Egirl presents it all in the context of a full visual outcomes model, she is in a position to have a rich discussion with her funder about what it makes sense for her to report on and measure and what it does not make sense for her to report on at the moment.
3. The reporting relationship is therefore much more intelligent and dynamic. It may be that this year it does not make sense to set a target on something, but maybe next year when some data is in from this year it will make more sense. Because the funder has much more information to go on in terms of what it is, and what it is not, possible to measure he or she should be much more relaxed regarding M&Egirl’s refusal to provide particular targets because either they cannot be set yet, or because it does not really make any sense to set them as targets.
4. The relationship between monitoring and evaluation can also immediately be illustrated using this visual approach. This is very important because it may be that a piece of evaluation can assist a funder have a clearer view of an aspect of a program which then makes them more relaxed about monitoring a more limited set of indicators and targets. In a case like M&Egirl’s progam, it is often very difficult to have an effective way of knowing whether or not clients are improving because of an intervention, just going on the basis of monitoring information. There can, of course, in some cases be monitoring of indicators of improvement (and this is usually a good idea) but these do not really establish whether or not the program is helping clients improve (it may just be a natural process of improvement for instance after any incidence of violence (the issue M&Egirl’s program is focused on) being over. Check out again the Five Building Blocks Diagram to see how such indicators are not-necessarily attributable indicators - their improvement cannot be definitively attributed to the program in question. In the mock-up you can see that there is an evaluation question ‘Are women entering the service receiving adequate interventions”? It is proposed that this is answered as a one-off evaluation exercise by an expert in the field doing a peer review visit to the program and assessing the likelihood that it is being effective for the women involved. This takes the pressure off just the indicator monitoring as having to, somehow, establish that the program is having an effect.
5. If the funder thinks that this is all too much for them to go through, they can commission someone to peer-review the outcomes model and the indicators for some assurance that M&Egirl is leaving no stone unturned in her effort to monitor the program.
6. Throughout working with the funder using this visual model, M&Egirl would be discussing with the funder about how much it is going to cost to monitor any indicator and answer any evaluation question. This forces the funder to be in the position of working out how they want to spend the monitoring and evaluation resources they have available on this particular program.
7. There are lots of other things that can be done with the visual outcomes model once the initial investment has been made in building it, e.g. planning economic evaluation, putting evidence in behind the links between steps, reporting evaluation results (see the Easy Outcomes website for ideas on this). M&Egirl can drill down further into the model to elaborate aspects of what is happening in the program. The funder can, if they wish, further develop the upper reaches of the model showing how the particular programs they are funding relate to their higher-level outcomes as a funder (see an example here of mapping projects onto a common set of outcomes).
So that is my suggestion for how we should deal with the targets and indicator question. It is likely that in the first instance M&Egirl’s funder is going to have their systems for how they want to collect monitoring information. But it may be that M&Egirl could build a visual model like this and initially just use it to make points in a meeting with her funders. She would probably have to keep using their system for at least a while, but it may be that in due course they see the merits of the richness this visual outcomes model approach can provide.
It is a somewhat more intensive approach to the dialogue between funders and providers than providers just emailing a list of indicators or targets to funders. However, as we’ve seen from M&Egirl’s comment and the experience of everyone else who has worked in such situations, the traditional systems all end up in endless, often incoherent and frustrating discussions anyway. Why not move to having a discussion around a visual model which is capable of representing a fair amount of the real complexity in the world around monitoring and evaluation rather than continuing to attempt to prop up the dream world as imagined in the Perfect Knowledge Fallacy.
Paul Duignan, PhD
Comments»
no comments yet - be the first?