Potential challenges to Systematic Outcomes Analysis July 23, 2007
Posted by Paul Duignan in : Systematic Outcomes Analysis, Evaluation planning , trackbackSystematic Outcomes Analysis claims to provide a standardized approach to outcomes, monitoring and evaluation planning [see my previous post Can outcomes, monitoring and evaluation planning be standardized?]. What are the challenges people are likely to make to this claim and can they be answered? I’ve set out some of the major potential challenges below and provided some thoughts on each of them:
1. It is not possible to have a standardized approach to outcomes, monitoring and evaluation planning, every situation is unique.
This is the argument that evaluation is a ‘craft’ needing a skilled evaluation planner to tailor an evaluation to fit the unique situation. How can a ‘cook-book’ standardized approach do justice to the complexity of real world programs. This challenge should not be accepted until it’s been proved to be true. The best way to try to prove it is to try out a system like Systematic Outcomes Analysis to see if it does, in fact, fail when being used to plan particular types of evaluations. Aspects of the system have already been applied in a number of situations without any difficulty, applying it in a diverse range of programs in the future will help us find out if there is anything in this challenge. Of course Systematic Outcomes Analysis just provides a framework of the basic decisions which need to be made in any evaluation. It is structured around the designer identifying evaluation questions and working out how they are going to answer them. As a result, there is no reason why the designer can’t use as much craft and creativity as they like in working out how they are going to answer the evaluation questions they have decided on.
.
2. Any standardized approach to evaluation will benefit one philosophy or evaluation paradigm over another and, given the diversity of evaluators’ paradigm postions, it will be rejected by many.
Systematic Outcomes Analysis wants to provide a generic framework for any type of evaluation philosophy or paradigm. While it provides a basic set of building blocks for any outcomes, monitoring or evaluation system, it allows plenty of room for a designer to just select the building blocks they want to use. It’s true that that the designer organizes their plan around an outcomes model so that you can map indicators and evaluation questions back onto the model. However, designers who don’t like outcomes models because they just want a strict ‘black-box’ evaluation design (looking only at outcomes rather than at the processes that lead to them) can still build their plan around a very simple outcomes model limited to the intervention and its final outcomes. Since Systematic Outcomes Analysis bases evaluation planning on defining and answering evaluation questions, it is possible for any evaluation question at all to be asked and for any method to be proposed for answering that question. Therefore the approach should not limit any evaluator working from any particular paradigm from being able to quickly set out what it is that they are planning to do.
3. Particular paradigms distinguish between different types of knowledge claim (e.g. the difference between high level outcome attribution claims and non-outcome (e.g. process evaluation) claims). A generic approach will inevitably force this distinction to be lost.
In the current climate at least, it is essential that a standardized framework avoids being so generic that it doesn’t let different paradigms make their strong claims about the type of outcomes and evaluation knowledge they think they are producing. Systematic Outcomes Analysis has been designed to allow such claims to be made. However, because it’s attempting to provide a generic framework, it doesn’t take a position on these claims, it simply allows the claims to be made so that they may be accepted or rejected by others. A major claim made by some is that certain types of experimental outcome evaluation design provide much stronger knowledge claims than other types of evaluation (e.g. process evaluation). In Systematic Outcomes Analysis, designers who hold this belief are allowed to differentiate such types of high level outcome evaluation from other types of evaluation. However, this is balanced within Systematic Outcomes Analysis by the fact that provision is made for other designers to reject such knowledge claims in regard to particular approaches because they regard them as not appropriate for philosophical, paradigmatic or other reasons, or because they are not feasible and/or affordable. This ‘right of refusal’ allows room firstly for those who reject all experimentalism as unable to do justice to measuring the outcomes of programs involving humans. Secondly, it also allows room for those who, while they have no problem with using experimental outcome evaluation designs such as randomized controlled experiments where they are appropriate, strongly argue that in many cases (e.g. where you cannot establish a credible non-treatment control) they are likely to produce weaker knowledge claims than the use of other alternative methods of evaluation.
4. Isn’t a positivist experimentalist bias hardwired into Systematic Outcomes Analysis due to the separate treatment of the list of seven possible outcome evaluation designs starting with ‘True experimental design.’
No matter what philosphical or paradigmatic position evaluators may hold, the current reality is that many stakeholders outside evaluation view evaluation as simply consisting of experimental outcome evaluation designs. In addition, the current international trend in evidence, results and outcomes focused practice is creating demand for ‘robust outcome evaluation studies’. Finally, in some situations, e.g. US educational research, particular experimental evaluation designs are being mandated for some pools of funding. For any standardized approach to evaluation planning to be useful in this institutional context, it’s important that it allows the designer to clearly specify the type of evaluation which they are planning to do, otherwise it will be regarded as irrelevant and useless by key and powerful stakeholders. However, it is equally important that such a standardized approach allows for a strong and clear case to be made against restricting the scope of ‘robust’ ‘credible’ or ‘relevant’ evaluation to a narrow limited set of experimental outcome evaluation designs.
Systematic Outcomes Analysis tries to avoid falling into the conceptual trap of juxtaposing experimental evaluation designs against all others. It does this by distinguishing outcome and other type of evaluation designs not on the basis of their methodology (e.g. experimental or other) but on the basis that these outcome evaluation designs are making a strong claim that they will say something about what has caused high level outcomes to change in a way that other types of evaluation are not. In its list of seven outcome evaluation designs, Systematic Outcomes Analysis, allows both traditional quantitative experimental type designs, but also other qualitative designs, rejected by strict experimentalists but accepted by some stakeholders in some situations as providing knowledge claims about causation in regard to outcomes. Systematic Outcomes Analysis leaves it up to designers and their stakeholders to decide which of these designs, if any, they regard as being appropriate, feasible and affordable.
Having allowed the planner to distinguish these types of design, Systematic Outcomes Analysis also ensures that the planner goes through each of these designs and has the opportunity to reject them on the basis of them not being appropriate, feasible and/or affordable. This approach has been designed to allow strong claims to be made where evaluation planners wish to make them. For instance a strictly quantitative evaluator is at liberty to reject the qualitative outcome designs as always being inappropriate while a qualitative evaluator is at liberty to reject all the quantitative outcome designs as always being inappropriate. However, them being forced to be transparent on this issue means that others can query, accept or reject their claims if they so wish.
This approach seems to me the only feasible way to build a generic approach to planning outcomes, monitoring and evaluation systems in a situation where some paradigms are making strong claims for certain evaluation methods and these are strongly backed by some instituational players.
Paul Duignan (outcomesblog.org)
Comments»
[…] of the way in which we set out outcomes, monitoring and evaluation plans. [See my next post Potential challenges to Systematic Outcomes Analysis for a discussion of the potential challenges Systematic Outcomes Analysis needs to be able to […]