Can an exhaustive list of impact evaluation designs be developed, or is my mission on this futile? August 27, 2009
Posted by Paul Duignan in : Impact evaluation, Evaluation debates, Attribution, Outcomes systems architecture, Outcomes theory, Evaluation planning, Using the approach, Doing evaluation more efficiently, Easy Outcomes , trackbackI have set out on a mission as a part of outcomes theory to attempt to develop an exhaustive list of impact/outcome evaluation designs - evaluation designs which make a claim that changes in high-level outcomes can be attributed to a particular intervention. If we could pull off developing such a list that most people are happy with, it would be very powerful. First it could be used in evaluation planning to work out if all of the possible impact evaluation designs had been assessed for their appropriateness, feasibility and/or affordability. At the moment I think that almost every evaluation planner walks around wondering if there is some sort of impact evaluation design they have not considered.
Obviously this would also assure evaluation funders and stakeholders that an exhaustive analysis had been done of what is possible in regard to impact evaluation within any evaluation plan, particularly if it was in a format that could be quickly peer reviewed. The more standard a generally accepted list we have, if properly documented, the quicker if would be for other people to peer review the evaluation planners analysis of what impact evaluation is possible. Examples of such an analysis can be found a the end of this article here.
Secondly, such an exhaustive list of impact evaluation designs can be used to set up a more standardized approach to setting out evaluation plans which is what I am trying to do in the Easy Outcomes approach. If we can reserve the ‘impact evaluation’ section to just one part of such a standardized approach to developing evaluation plans (e.g. in a DoView Visual Evaluation Plan like this one here). then people with less impact evaluation experience can work up the rest of the evaluation plan and just go to an impact evaluation expert to talk about the impact evaluation section. I think that this is a much more feasible approach to promoting impact evaluation than expecting everyone developing an evaluation plan to be an expert at impact evaluation - actually, it’s rather complex stuff. An article on developing such a standardize approach is here.
Lastly, if we can do it, such an approach means that we can be clear when impact/outcome evaluation is not appropriate, feasible and/or affordable and so we can then look as part of strategic evaluation decision-making about where to spend evaluation resources and the mix of other types of evaluation we may want to use (see here for other types of non-impact evaluation - formative and process). We can also think about what our overall evaluation scheme is. It may be impact evaluation only in a piloting or demonstration project stage and no impact evaluation on full roll-out of the program (just best practice sharing). (See here for an article on full roll-out impact evaluation versus just on piloting and see here for the article mentioned above about about deciding when impact evaluation is, and is not, appropriate, feasible and/or affordable).
Anyway, I have discussed attempting to build such a list with a number of people, some who are skeptical about the idea of being able to do so. Others find what I have come up with really useful and are excited by what I am trying to do. At the moment there is some interesting discussion about what the list should include, particularly around the issue of how to deal with more qualitative designs - I am trying to have a list which includes both more quantitatively orientated and more qualitatively orientated designs. If you have a moment, check out the discussion at the bottom of the Knol article where I am trying to identify the exhaustive list of impact evaluation designs and I would be interested in you adding any comments if you have an angle on it. The article with comments at the bottom is at:
Duignan, P. (2005-2009). Seven possible impact/outcome evaluation design types. Outcomes Theory Knowledge Base article No. 209). (http://knol.google.com/k/paul-duignan-phd/seven-possible-outcomeimpact-evaluation/2m7zd68aaz774/10).
Paul Duignan, PhD. (Follow me on my Outcomes Blog; Twitter; or via my E-Newsletter).
Comments»
no comments yet - be the first?