Being sector-centric not program-centric in deciding on program evaluation priorities May 22, 2009
Posted by Paul Duignan in : Impact evaluation, Doing evaluation more efficiently, Evaluation planning , trackbackI have been blogging less in the last week or so due to going on holiday - blogging will still be less frequent for a week or so. I was recently involved in assessing a set of projects summaries to help determine which of them should be selected for more intensive evaluation input. This was not being done directly to determine project evaluation priorities, however the exercise reminded me once again of the general issue of how we determine what types of evaluation should be undertaken for particular projects. My blog comments below are about the general issue rather than the particular exercise of project selection I was recently involved in. The set of information we typically use to work out what type of evaluation should be undertaken for a project is information such as the nature of the project, the proposed evaluation questions, and the proposed evaluation budget. I think that we need more than this when determining evaluation priorities.
What is missing for me is any sense of where a project sits within the sector in which it is located. I have been advancing the argument that we should be reframing program evaluation as part of collecting strategic information for sector decision-making. If we explore the implications of this argument, in addition to the type of information we often have on the individual projects (we can call this program-centric information), we also need information about where particular programs sit within their sector (sector-centric information). It is likely that this information cannot just be provided by a program itself, it is overview information which usually would appropriately be collected and collated by some sort of central or coordinating organization or collaborative mechanism.
The type of information we need if we take a more sector-centric approach is: is this a novel program for the sector or is this a program which has been running in this sector for a considerable period of time? What previous evaluation information is there on the mechanism of the intervention being used, what does that evaluation information say? What is the interaction between this program and other activity in the sector? What type of sector resource allocation decisions relating to projects like this one, are likely to be made about the time the evaluation results are produced? How scalable is the program in the sector in which is it operating?
This sort of information can assist in setting project evaluation priorities. For instance if a program is using an intervention mechanism which has been used often in the past in a sector and which has been positively evaluated, it may be better to restrict monitoring and evaluation activity to just monitoring that best practice is being implemented in the program rather than attempting impact evaluation (establishing that a high-level impact has taken place).
Obviously, collecting and working with this type of sector information is going to be a much more resource intensive activity than just looking at the characteristics of a single program when thinking about what evaluation is appropriate for the program. However I think that over time we should be working towards such aggregations of evaluation information at the sector level, of course the large meta-analysis and evidence-based practice data collections (such as the Cochrane Collaboration in health and the Campbell Collaboration in social programs are heading in this direction). Aggregations of current evaluation findings from a sector can then be used to identify priority evaluation questions and these can then be used to inform decision making about the priority for evaluating a particular project.
Of course funders and other stakeholders are often trying to take sector information into account when thinking about project evaluation priorities. What I am suggesting is trying to push this type of thinking further and ultimately to formalize it more into sector evaluation plans which identify priorities for future monitoring and evaluation within a sector.
Another insight I had when doing the reviewing exercise I mentioned at the start of this blog was what is happening in regard to expectations about what type of evaluation can be done. Given the increased pressure for higher-level impact evaluation I think that we are moving into a climate where providers feel that they should be offering to undertake impact evaluation on all their projects. Sometimes this is being proposed in the face of very small budgets for evaluation. This runs the risk of what I call pseudo-impact evaluations being done where a small level of resources available for evaluation results in an impact evaluation which is not sufficiently robust to actually establish that an effect has occurred. Often it may be better for evaluation resources to be spent on a smaller number of more robust impact evaluations than attempting impact evaluation for a wide range of programs. I talk about this issue here.
Paul Duignan, PhD
Outcomes and Evaluation Blog (OutcomesBlog.org)
Comments»
no comments yet - be the first?