Impact evaluation’s day in the sun (Part I) November 30, 2008
Posted by Paul Duignan in : Attribution, Evaluation debates, Evaluation planning, Impact evaluation , trackback
When you’ve been around for a while you see things come and go. Impact evaluation – evaluation looking at the longer-term (or ‘higher-level’ outcomes within an visual outcomes model) is currently enjoying a resurgence (sometimes its just called plain outcome evaluation). I’ve been attending a number of international evaluation conferences recently and had the pleasure of going to several workshops from experts on impact evaluation – Gary Henry from the University of North Carolina who ran a workshop at the United Kingdom Evaluation conference in Bristol and Charles Reichardt from the University of Denver ran a workshop on estimating the effects of interventions at the American Evaluation Conference in Denver. I thought both were excellent and showed the level of sophistication some of the experts thinking about impact/outcome evaluation are going to to do the job properly. I will blog in more detail about what they were saying a little later. However, these presentations were little oases of calm in a wider and more heated debate in evaluation circles about the resurgence of impact evaluation.
The potted history of impact evaluation as I see it is that in the early days it was seen as being what evaluation was all about, a lot of it was done in the U.S. in the 70′s. It cost a lot and its findings were not always implemented due to political and other reasons. In the 80′s and 90′s the focus shifted somewhat towards more qualitative approaches (not that the distinction qualitative/quantitative should be identified exactly with a process/impact evaluation distinction) with the push by Guba and Lincoln and Fourth Generation Evaluation towards more qualitative evaluation. Carol Weiss chipped away at a simplistic point of view about the dissemination of evaluation findings, pointing out that getting evaluation findings translated into practice is a much more complicated affair that just emailing them to busy policy makers and expecting them to implemented immediately. Michael Patton pointed out that we should be looking at the point of view of the users of evaluation and that impact evaluation is often delivered too late for some sets of users (e.g. those implementing a program).
What happened in the 2000′s is that a resurgence of interest in impact evaluation occurred. This was at its most assertive in decisions by some education research funders in the U.S. to only fund experimental and regression discontinuity impact designs (if you want more information on these designs see my description of these in the context of a list of possible impact/outcome evaluation designs). The overall trend towards more emphasis on impact evaluation is now being picked up in all areas of evaluation. It can be seen as part of the overall international emphasis on evidence-based practice – things like the Cochrane Collaboration in medicine and the Campbell Collaboration in social policy which are looking at all of the impact evaluation results and attempting to provide ‘what works’ summaries from all of the studies that have been done by using meta-analysis (a statistical way of summarizing the results from a number of quantitative impact evaluation studies of the same type of program).
Anyway, discussion about this trend was woven through the presentations at recent conferences like the European Evaluation Society Conference in Lisbon and the American Evaluation Association Conference in Denver. At the Lisbon EES conference I listened to a keynote presentation on impact evaluation in the international development sector – something of a hot topic at the moment. Towards the end there was a classic interaction which is typical of the way the debate about impact evaluation tends to go. The presentation had promoted the concept of doing impact evaluations, at the end a commentator made the point that there is a place for process evaluation, that impact evaluation may not be capturing all that is going on in a project. The comment sort of just hung there with no one really sure where we should go from there. I am sure that the impact evaluation enthusiasts would not have been daunted by this comment one little bit and that the process evaluation enthusiasts would have just see it as a declaration of their side of the debate with which they whole heartily agreed.
I think that the debate is structured in an unfortunately binary sort of way. Rather like two street fighting cowboys involved in a shoot out who are going to debate to the death as to whether impact evaluation or some other type (maybe process evaluation?) is going to be the only game in town. It is so reminiscent of the great qualitative versus quantitative debate in evaluation where we were equally going to decide on one side or the other. In fact one colleague who has been talking to Michael Patton about it says that he thinks that the current debate is just a re-run of the old qualitative versus quantitative debate dressed up in new clothes.
If we ever want to get beyond either the two sides shooting it out (which is only going to appeal to extremists on either side) or just making the somewhat bland statement that there is a place for all kinds of evaluation without any more guidance than that, we need to break the debate up into its sub-debates and have a look at each of them as potentially separate issues. I’ll start doing this in my blog posting tomorrow. (Part II tomorrow).
Paul Duignan, Phd
(OutcomesBlog.org)
Comments»
[...] the previous posting Part I first). Progressing the debate around impact evaluation means unpacking some of the different [...]
[...] Read Part I and Part II first]. Now that I have got to Part III of this posting on the impact evaluation [...]